{"name":"split-classify-extract","title":"Split, Classify and Extract","keyPoints":["Only GPT-6 Luna and GLM-5.3 Flash on Z.ai get 25 of 25 on the two sets and 15 of 15 on the court filings; on the filings GPT-6 Luna costs $0.0336 against $0.0624 and takes 64 seconds against 394.","Ling 3.0 Flash and Claude Haiku 5.5 leave one court filing, a podcast transcript admitted as a trial exhibit, under no requirement.","On 105 pages Claude Haiku 5.5 costs 3.9 times GPT-6 Luna, $0.0932 against $0.0237: its one prompt over 100,000 tokens paid five times Haiku's usual price, and Claude counts 56% more input tokens for the same requests."],"tables":[{"heading":"Ten text models, 25 checks each: 6 documents cut out, 6 matched to their requirement, 13 values read","columns":["Model","Price in / out","Split","Classify","Extract","Tokens in / out","Cost","Seconds"],"rows":[["Ling 3.0 Flash (Novita)","$0.06 / $0.18","6 of 6","6 of 6","13 of 13","78,436 / 17,368","$0.0078","64"],["GPT-6 Luna (OpenAI)","$0.10 / $0.50","6 of 6","6 of 6","13 of 13","71,521 / 5,671","$0.0100","57"],["Claude Haiku 5.5 (Anthropic)","$0.10 / $0.50","6 of 6","6 of 6","13 of 13","111,993 / 10,776","$0.0166","49"],["GLM-5.3 Flash (Z.ai)","$0.15 / $0.50","6 of 6","6 of 6","13 of 13","73,274 / 20,976","$0.0215","321"],["GLM-5.3 Flash (Novita)","$0.15 / $0.50","4 of 6","6 of 6","13 of 13","73,252 / 14,830","$0.0184","193"],["gpt-oss-120b (Novita)","$0.05 / $0.25","3 of 6","5 of 6","13 of 13","72,349 / 10,890","$0.0063","65"],["GLM-4.7 FlashX (Z.ai)","$0.07 / $0.40","3 of 6, 3 not answered","3 of 6","13 of 13","48,277 / 25,519","$0.0136","699"],["Nemotron 3 Nano 30B (Novita)","$0.05 / $0.20","1 of 6, 3 not answered","4 of 6","13 of 13","51,548 / 23,887","$0.0074","116"],["GLM-4.7 Flash (Novita)","$0.07 / $0.40","1 of 6, 3 not answered","2 of 6","13 of 13","48,255 / 29,726","$0.0153","640"],["GLM-4.7 Flash (Z.ai)","free","0 of 6, 6 not answered","0 of 6, 2 not answered","8 of 13","21,349 / 14,349","$0.0000","505"]],"note":"Price: US dollars a million tokens. Cost: the providers' token prices. Seconds: each model's calls added up. Classify: the requirement each document matches; its kind is not counted. Not answered: the provider gave no answer. Extract asks Claude Haiku 5.5 to cite its lines, the others for the values alone."},{"heading":"Split and Classify on one file of six documents, 105 pages","columns":["Model","Split","Classify","Tokens in / out","Cost","Seconds"],"rows":[["Ling 3.0 Flash (Novita)","6 of 6","6 of 6","224,784 / 17,095","$0.0166","87"],["GPT-6 Luna (OpenAI)","6 of 6","6 of 6","203,776 / 6,587","$0.0237","54"],["Claude Haiku 5.5 (Anthropic)","6 of 6","6 of 6","318,004 / 13,273","$0.0932","56"]],"note":"Split and Classify only: no model is asked to cite. Classify: the requirement each document matches; its kind is not counted. Cost: Claude Haiku 5.5 pays $0.50 / $2.50 a million for a prompt over 100,000 tokens, as one of its calls here was: 119,461 tokens."},{"heading":"Classify with the case's own list: 15 court filings of one civil trial, 236 pages","columns":["Model","Classify","Tokens in / out","Cost","Seconds"],"rows":[["GPT-6 Luna (OpenAI)","15 of 15","314,170 / 4,454","$0.0336","64"],["GLM-5.3 Flash (Z.ai)","15 of 15","317,641 / 29,449","$0.0624","394"],["Ling 3.0 Flash (Novita)","14 of 15","356,201 / 14,278","$0.0239","78"],["Claude Haiku 5.5 (Anthropic)","14 of 15","432,330 / 11,495","$0.0490","61"]],"note":"Classify: the requirement each filing matches. The list has one for every filing, and its kind follows from it. Claude Haiku 5.5 ran later the same day, on the same inputs."}],"scoringMethod":"Each model ran once on each set: two public sets of documents, a 105-page file (three models), and the 15 filings of one civil trial with that case's own list. The grader, a program, scored each answer against ground truth Claude wrote from the pages; no person has checked it yet.","log":[{"date":"261015","line":"Added Classify on 15 court filings of one civil trial, served the case's own list: GPT-6 Luna and GLM-5.3 Flash on Z.ai 15 of 15, Claude Haiku 5.5 and Ling 3.0 Flash 14 of 15."},{"date":"261015","line":"Corrected: four models get 25 of 25, GLM-5.3 Flash on Z.ai as well; Claude Haiku 5.5's cost on 105 pages has two causes, neither of them citing; Classify counts the requirement only."},{"date":"261015","line":"New board. Ling 3.0 Flash, GPT-6 Luna and Claude Haiku 5.5 tie at 25 of 25; Ling 3.0 Flash costs least."}],"updatedAt":"2026-10-15T19:27:36.519Z"}