Hacker News

SEPTEMBER 27, 2023

This new primitive builds on the foundation set by Deno KV, and is available today in the Deno JavaScript runtime and Deno Deploy. Introducing Deno Queues - zero config, scalable messaging with a guaranteed at-least-once delivery.

181

181

Hacker News

DECEMBER 18, 2023

Gos hófst um fjóra kílómetra norðaustan Grindavíkur, norðan Sundhnúks í Sundhnúkaröðinni, klukkan 22:17 í kvöld.

144

144

Hacker News

JANUARY 27, 2024

At present, reducing resource consumption by compressing the KV cache is a common approach. We propose a new method, Lossless Compressed Memory Attention (LoMA), which allows for lossless compression of information into special memory token KV pairs according to a set compression ratio.

115

115

Hacker News

JANUARY 20, 2024

We also propose a segment KV cache policy to keep key/value of the request and response tokens in separate physical memory for effective device memory management, helping enlarge the runtime batch size and improve system throughput. We implement our LLM inference solution on Intel GPU and publish it publicly.

124

124

Hacker News

SEPTEMBER 19, 2023

Comments (..)

104

104

Julien Simon

MARCH 11, 2024

In this video, we zoom in on optimizing LLM inference, and study key mechanisms that help reduce latency and increase throughput: the KV cache, continuous batching, and speculative decoding, including the state-of-the-art Medusa approach.

Deep Learning 52

Deep Learning Deep Learning 52

Towards AI

NOVEMBER 17, 2023

Lower computational complexity By using KV cache, the computational cost of calculating tensor Key and Value in each step of MQA is 1/h of MHA, where h represents the number of heads. MQA has a relatively small KV cache that can fit into the cache (SRAM).

AI 111

AI AI Machine Learning Machine Learning 111

Towards AI

JANUARY 13, 2024

LLMLingua uses GPT2-small and LLaMA-2-7B to decrease the prompt size by 20x TLDR: If you want to U+1F4B0 Save Cost by reducing both prompt and generation lengths.U+1F4DD 1F4DD Extend Context Support beyond Model limits of the APIs and ModelsU+1F4DC Mitigates the “lost in the middle” issue and boost overall performance.U+1F575️

AI 103

AI AI Algorithm Data Science 103

Hacker News

JULY 21, 2023

llama_init_from_file: kv self size = 780.00 llama_init_from_file: kv self size = 780.00 llama_init_from_file: kv self size = 780.00 llama_init_from_file: kv self size = 780.00 llama_init_from_file: kv self size = 780.00 llama_init_from_file: kv self size = 780.00 MB (+ 3124.00 MB per state). MB (+ 3124.00

94

94

Hacker News

FEBRUARY 13, 2023

Comments (..)

94

94

Towards AI

FEBRUARY 5, 2024

We’ll add a special token/character to denote the start and end of the word bg_pairs = dict()for word in words: word = [' '] + list(word) + [' '] for ch1, ch2 in zip(word, word[1:]): bg_pair = (ch1, ch2) bg_pairs[bg_pair] = bg_pairs.get(bg_pair, 0) + 1 Printing out the top 10 bigram pair based on their frequency sorted(bg_pairs.items(), (..)

AI 94

AI AI Data Science Machine Learning 94

ODSC - Open Data Science

OCTOBER 13, 2023

It addresses two major challenges in deploying LLMs: caching previous tokens’ key and value states (KV) consumes extensive memory, and popular LLMs cannot generalize to longer texts than the training sequence length. DocsGPT DocsGPT is an open-source solution that streamlines the process of finding information in project documentation.

Data Science 52

Data Science AI AI Artificial Intelligence 52

Hacker News

JUNE 12, 2023

Especially for long generations this makes a large difference because the KV cache is still CPU only on master and gets larger as the context fills up. I also added two special kernels for doing matrix vector multiplication with permuted or not contiguous inputs; they are used in conjunction with the KV cache. Changes to ggml.c:

98

98

AWS Machine Learning Blog

APRIL 8, 2024

This reduces the size of the KV cache on GPU memory, allowing for greater concurrency, and improving price-performance. Different KV cache sharding strategies are available to suit different types of workloads. It allows you to insert and remove KV cache rows from new requests while you’re handing batched inference.

AWS 84

AWS Deep Learning Deep Learning Machine Learning 84

AWS Machine Learning Blog

OCTOBER 11, 2023

Forward pass, activations, and the KV cache Given an input sequence of tokens, they are run in a forward pass across all the layers of the LLM (like Falcon) to generate the next token. These cached key and value tensors are often referred to as the KV cache.

AWS 95

AWS Machine Learning Machine Learning ML 95

Mlearning.ai

JULY 5, 2023

Table of Contents Introduction to Kivy Benefits of Kivy Installation and Setup Creating a Basic Kivy App Kivy Widgets and Layouts Event Handling and Properties Kivy Language (KV) for UI Design Kivy Screen Manager Integrating with Other Python Libraries Deploying Kivy Apps to Different Platforms 1. kv extension. kv extension.

Python 52

Python ML ML Machine Learning 52

Towards AI

DECEMBER 28, 2023

However, as the context window or batch size increases, the memory cost associated with the size of the key-value cache(kv cache) in the multi-head attention(MHA) model significantly increases.

AI 105

AI AI Machine Learning Machine Learning 105

AWS Machine Learning Blog

JANUARY 29, 2024

KV caching and device memory Standard transformer attention mechanisms compute attention for each new token against all previous tokens. This is called this the KV cache , and it grows with batch size and sequence length. The following formula is a rough approximation for the maximum KV cache size.

ML 95

ML ML Machine Learning Machine Learning 95

AssemblyAI

AUGUST 3, 2023

. * * - Run `npm run dev` in your terminal to start a development server * - Open a browser tab at [link] to see your worker in action * - Run `npm run deploy` to publish your worker * * Learn more at [link] */ export interface Env { // Example binding to KV.

Database 97

Database 97

Towards AI

APRIL 8, 2024

Furthermore, by substituting some Transformer layers with Mamba layers, Jamba significantly reduces the size of the key-value (KV) cache needed for processing, achieving up to an eight-fold decrease compared to traditional Transformers.

Machine Learning 96

Machine Learning Machine Learning AI AI 96

AWS Machine Learning Blog

SEPTEMBER 25, 2023

These cached key and value tensors are often referred to as the KV cache or attention cache. As per the paper vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention , the KV cache takes up to 1.7 As a result, efficiently managing the KV cache presents a significant challenge. GB for a single sequence in Llama 13B.

AWS 94

AWS Machine Learning Machine Learning Deep Learning 94