July 23, 2024

July 23, 2024

July 23, 2024

---

Follow up on the ChatGPT can’t summarise post.

RULER: What's the Real Context Size of Your Long-Context Language Models?

[2404.06654] RULER: What's the Real Context Size of Your Long-Context Language Models?

The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequ...

Benchmarking Large Language Models for News Summarization

https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00632/119276/Benchmarking-Large-Language-Models-for-News

Anonymous: (let me know if you want credit)

The gist is that LLMs absolutely can summarize but have issues with context size and coherence on length.
Chunking and recursive summarization are both valid and recognized strategies and there are other benchmarks specifically designed to measure summarization effectiveness(which escape me right now)

And from the discussion at hacker news:

When ChatGPT summarises, it does nothing of the kind | Hacker News

https://news.ycombinator.com/item?id=41027658

Am I blind or is there no mention at all of the GPT model he used?
The author states his conclusions but doesn't give the reader the information required to examine the problem.
- Whether the article to be summarized fits into the tested GPT model's context size
- The prompt
- The number of attempts
- He doesn't always state which information in the summary, specifically, is missing or wrong
For example: "I first tried to let ChatGPT one of my key posts (...). ChatGPT made a total mess of it. What it said had little to do with the original post, and where it did, it said the opposite of what the post said." He doesn't say which statements of the original article were reproduced falsely by ChatGPT.
My experience is that ChatGPT 4 is good when summarizing articles, and extremely helpful when I need to shorten my own writing. Recently I had to write a grant application with a strict size limit of 10 pages, and ChatGPT 4 helped me a lot by skillfully condensing my chapters into shorter texts. The model's understanding of the (rather niche) topic was very good. I never fed it more than about two pages of text at once. It also adopted my style of writing to a sufficient degree. A hypothetical human who'd have to help on short notice probably would have needed a whole stressful day to do comparable work.

---

Cybercrooks crafting solo careers in wake of ransomware takedowns

Interesting development (if true):

https://www.theregister.com/2024/07/22/europol_says_ransomware_takedowns_make/

---

---

---

---

---

---