Author SHA1 Message Date
CedricandClaude Opus 5.5 1a3442ea02 Merge ticket/3-fun-section into feature/source-kinds (#3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:58:43 +02:00
CedricandClaude Opus 5.5 1048e4e384 Merge feature/source-kinds into ticket/3-fun-section (#3)
Keeps the Learning catalog from #6 and puts the Fun group after it, in the
catalog and the README. Adds the end-to-end check #4 left for Fun: a custom
Fun feed added on the Sources tab shows up in the Learn tab's Fun section,
not in Today.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:58:17 +02:00
CedricandClaude Opus 5.5 84cb11e4dd Hide paid Substack newsletter posts with the paid filter (#7)
Substack cuts paid-subscriber posts short in the feed and ends their
content with a "Read more" link to the post itself. PaywallService.isPaid
now recognizes that ending, so "hide paid articles" hides these posts in
Today and the Learn tab and counts them as hidden, without a page request.
Free posts that ask readers to subscribe or end with the subscribe widget
are not affected.

Tests use trimmed real items from the Interconnects and Ahead of AI feeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:57:48 +02:00
CedricandClaude Opus 5.5 8240a767ef Add the Fun section to the Learn tab with comic-friendly reader view (#3)
The Learn tab gets a Fun section below Learning with the last 3 days of
Fun articles. Fun articles stay out of Today and the Briefing. New curated
Fun sources: xkcd and The Daily WTF (on by default) and r/ProgrammerHumor
top of the day (off).

Reader view for Fun articles shows the picture whole in the body instead
of as a 16:9 banner cropped to fit, and doesn't call a comic without text
a teaser. When the page gives no picture (Reddit blocks reader view), the
feed's image is shown. The extractor now keeps an image's title text as a
caption (xkcd's last joke), images in blockquotes and behind in-page
anchors (The Daily WTF's Error'd screenshots), drops image maps and
license boxes, and on a text tie prefers the page with more blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:56:42 +02:00
CedricandClaude Opus 5.5 4f035e4bb4 Merge ticket/6-learning-catalog into feature/source-kinds (#6)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:51:33 +02:00
CedricandClaude Opus 5.5 b634118ed8 Add the curated Learning sources (#6)
Fifteen English Learning sources join ByteByteGo in the catalog: System
Design One, The Pragmatic Engineer (blog), Martin Fowler, Julia Evans,
Dan Luu, The Cloudflare Blog, Netflix Technology Blog, Engineering at
Meta, GitHub Blog: Engineering, Latent Space, Import AI, Interconnects,
Ahead of AI and Lilian Weng (on by default), and InfoQ (off).

Tests pin the Learning catalog, check that existing installs get the
new on-by-default Sources and that the Learn tab shows their posts from
the last 14 days. A live test times a full refresh. The README gets a
Learn feature bullet and a Learning sources table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:50:50 +02:00
CedricandClaude Opus 5.5 f205452b25 Group the Sources tab by kind and pick a kind when adding a feed (#4)
The Sources tab shows News, Learning and Fun sections, each split by
language; kinds or languages without Sources get no header. The add-feed
dialog has a kind picker that defaults to News.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:50:43 +02:00
CedricandClaude Opus 5.5 a6b8d8b537 Merge feature/source-kinds into ticket/2-source-kinds-learn-tab (#2)
Brings in the News catalog update (#5). ByteByteGo joins the expected
on-by-default sets for updated installs, and the catalog test checks the
on-by-default set among News sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:45:55 +02:00
CedricandClaude Opus 5.5 64c4a11105 Add source kinds and the Learn tab with Learning sources (#2)
Every Source now has a kind: News, Learning or Fun. Curated Sources default
to News, and custom Sources persist their kind (ones saved before default
to News). Each kind has its own time window: News follows the 12/24/48 h
setting, Learning reaches back 14 days, Fun 3 days.

A new Learn tab between Briefing and Sources lists Learning articles with
the same paid/sponsored filters and reader view as Today. It is built from
a list of kinds so a Fun section can follow. Today and the Briefing only
use News. ByteByteGo is the first Learning source, on by default.

The home shell names its tabs instead of using hard-coded indexes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:44:52 +02:00
27 changed files with 1520 additions and 79 deletions

No files matched your search

+53 -2
View File
@@ -8,6 +8,9 @@ or write a daily briefing. It's meant to replace the n8n → Discord workflow.
- **Today**: all articles from the enabled sources in the last 12/24/48 h, newest
first. Filter by language or source, pull to refresh.
- **Learn**: articles from the enabled Learning sources (engineering blogs,
system design explainers, AI research write-ups) from the last 14 days, newest
first. Learning sources aren't part of the Briefing.
- **Reading in the app**: tapping an article opens a clean **reader view** (text,
headings, lists, quotes, images) without ads or cookie banners. The globe icon
switches to the **original page** in an in-app WebView, for paywalls or sites
@@ -22,7 +25,9 @@ or write a daily briefing. It's meant to replace the n8n → Discord workflow.
- **Paid articles and ads hidden**: heise+ and Golem `(g+)` are recognized by
their title; SPIEGEL+ and t3n Pro by a marker in the article page, which the app
checks once per new article in the background (it reads only until the marker,
and remembers results for 14 days). Sponsored posts (`Anzeige:`) are hidden too.
and remembers results for 14 days). Paid-subscriber posts from Substack newsletters
(Interconnects, Ahead of AI, …) are recognized in the feed itself, where Substack
cuts them off with a "Read more" link. Sponsored posts (`Anzeige:`) are hidden too.
The feed header shows how many were hidden; both filters can be turned off.
- **Ad and cookie-banner blocking** in the original-page view: requests to ~80
ad/tracker hosts and the common consent managers (Sourcepoint, OneTrust,
@@ -34,7 +39,8 @@ or write a daily briefing. It's meant to replace the n8n → Discord workflow.
valid for a day) and reuses the cookie for reader view and the in-app page.
With it off, the app shows the teaser and an "Open website anyway" option.
- **Sources**: 23 curated feeds (13 German, 10 English), toggle each one, or add any
RSS/Atom feed (it's validated before being added).
RSS/Atom feed as a News, Learning or Fun source (it's validated before being
added). The tab groups Sources by kind, then by language.
- **Dark mode**: System / Light / Dark in Settings (applies instantly). The
original-page view follows it too: when the phone is dark, sites use their own
dark design or Android darkens them; when only the app is dark, the app inverts
@@ -180,6 +186,51 @@ page. Hacker News and Lobsters link to articles all over the web, so reader view
depends on the linked site. Techmeme links to its own summary page, which only
has the teaser; the original page links on to the full story.
## Learning sources
Evergreen reading for building skills: research, engineering depth and system
design rather than product news. The Learn tab shows their posts from the last
14 days. All are English; ★ marks the ones enabled by default.
| | Source | Feed | Focus |
|---|---|---|---|
| 🇬🇧 ★ | ByteByteGo | `blog.bytebytego.com/feed` | System design explained with diagrams |
| 🇬🇧 ★ | System Design One | `newsletter.systemdesign.one/feed` | System design case studies of real systems |
| 🇬🇧 ★ | The Pragmatic Engineer (blog) | `blog.pragmaticengineer.com/rss/` | How big tech and startups build software (free posts) |
| 🇬🇧 ★ | Martin Fowler | `martinfowler.com/feed.atom` | Software architecture, refactoring, delivery |
| 🇬🇧 ★ | Julia Evans | `jvns.ca/atom.xml` | Systems, networking and Unix tools, explained plainly |
| 🇬🇧 ★ | Dan Luu | `danluu.com/atom.xml` | Long, data-driven essays on computers and engineering |
| 🇬🇧 ★ | The Cloudflare Blog | `blog.cloudflare.com/rss/` | Networking, security, incident write-ups |
| 🇬🇧 ★ | Netflix Technology Blog | `netflixtechblog.com/feed` | Distributed systems, data, streaming at scale |
| 🇬🇧 ★ | Engineering at Meta | `engineering.fb.com/feed/` | Infrastructure, AI, production engineering |
| 🇬🇧 ★ | GitHub Blog: Engineering | `github.blog/engineering/feed/` | How GitHub builds and runs GitHub |
| 🇬🇧 ★ | Latent Space | `www.latent.space/feed` | AI engineering: interviews and deep dives |
| 🇬🇧 ★ | Import AI | `importai.substack.com/feed` | Weekly AI research digest by Jack Clark |
| 🇬🇧 ★ | Interconnects | `www.interconnects.ai/feed` | Language model training and research |
| 🇬🇧 ★ | Ahead of AI | `magazine.sebastianraschka.com/feed` | LLM research explained by Sebastian Raschka |
| 🇬🇧 ★ | Lilian Weng | `lilianweng.github.io/index.xml` | Thorough surveys of machine learning topics |
| 🇬🇧 | InfoQ | `feed.infoq.com/` | Architecture and engineering talks, high volume |
Left out on purpose: The Pragmatic Engineer newsletter (mostly paid), One Useful
Thing (applied AI), vendor blogs (OpenAI, DeepMind, Hugging Face) and Chip Huyen
(inactive since January 2025). Some feeds are long (Dan Luu ~130 posts, Lilian
Weng ~50); the 14-day window keeps the list short.
## Fun sources
Tech humour for the Fun section of the Learn tab, which reaches back 3 days. Fun
articles never show up in Today or the Briefing. ★ = enabled by default.
| | Source | Feed | Focus |
|---|---|---|---|
| 🇬🇧 ★ | xkcd | `xkcd.com/atom.xml` | Webcomic of romance, sarcasm, math and language |
| 🇬🇧 ★ | The Daily WTF | `thedailywtf.com/rss` | IT horror stories, bad code and error messages |
| 🇬🇧 | r/ProgrammerHumor | `reddit.com/r/ProgrammerHumor/top/.rss?t=day` | The day's top programming memes on Reddit |
Reader view shows comics and memes whole instead of as a cropped banner: xkcd
with its title text underneath, The Daily WTF's Error'd with its screenshots.
Reddit blocks reader view, so a meme shows the picture from the feed.
## Project layout
```
+192
View File
@@ -200,6 +200,198 @@ const defaultSources = <NewsSource>[
language: SourceLanguage.en,
description: 'Security and malware news',
),
// --- Learning -----------------------------------------------------------
NewsSource(
id: 'bytebytego',
name: 'ByteByteGo',
feedUrl: 'https://blog.bytebytego.com/feed',
homepage: 'https://blog.bytebytego.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'System design explained with diagrams',
enabledByDefault: true,
),
NewsSource(
id: 'systemdesignone',
name: 'System Design One',
feedUrl: 'https://newsletter.systemdesign.one/feed',
homepage: 'https://newsletter.systemdesign.one',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'System design case studies of real systems',
enabledByDefault: true,
),
NewsSource(
id: 'pragmaticengineer-blog',
name: 'The Pragmatic Engineer (blog)',
feedUrl: 'https://blog.pragmaticengineer.com/rss/',
homepage: 'https://blog.pragmaticengineer.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'How big tech and startups build software (free posts)',
enabledByDefault: true,
),
NewsSource(
id: 'martinfowler',
name: 'Martin Fowler',
feedUrl: 'https://martinfowler.com/feed.atom',
homepage: 'https://martinfowler.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Software architecture, refactoring, delivery practices',
enabledByDefault: true,
),
NewsSource(
id: 'juliaevans',
name: 'Julia Evans',
feedUrl: 'https://jvns.ca/atom.xml',
homepage: 'https://jvns.ca',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Systems, networking and Unix tools, explained plainly',
enabledByDefault: true,
),
NewsSource(
id: 'danluu',
name: 'Dan Luu',
feedUrl: 'https://danluu.com/atom.xml',
homepage: 'https://danluu.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Long, data-driven essays on computers and engineering',
enabledByDefault: true,
),
NewsSource(
id: 'cloudflare-blog',
name: 'The Cloudflare Blog',
feedUrl: 'https://blog.cloudflare.com/rss/',
homepage: 'https://blog.cloudflare.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Networking, security and incident write-ups in depth',
enabledByDefault: true,
),
NewsSource(
id: 'netflix-techblog',
name: 'Netflix Technology Blog',
feedUrl: 'https://netflixtechblog.com/feed',
homepage: 'https://netflixtechblog.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Distributed systems, data and streaming at scale',
enabledByDefault: true,
),
NewsSource(
id: 'meta-engineering',
name: 'Engineering at Meta',
feedUrl: 'https://engineering.fb.com/feed/',
homepage: 'https://engineering.fb.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Infrastructure, AI and production engineering at Meta',
enabledByDefault: true,
),
NewsSource(
id: 'github-engineering',
name: 'GitHub Blog: Engineering',
feedUrl: 'https://github.blog/engineering/feed/',
homepage: 'https://github.blog/engineering/',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'How GitHub builds and runs GitHub',
enabledByDefault: true,
),
NewsSource(
id: 'latent-space',
name: 'Latent Space',
feedUrl: 'https://www.latent.space/feed',
homepage: 'https://www.latent.space',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'AI engineering: interviews and deep dives',
enabledByDefault: true,
),
NewsSource(
id: 'import-ai',
name: 'Import AI',
feedUrl: 'https://importai.substack.com/feed',
homepage: 'https://importai.substack.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Weekly AI research digest by Jack Clark',
enabledByDefault: true,
),
NewsSource(
id: 'interconnects',
name: 'Interconnects',
feedUrl: 'https://www.interconnects.ai/feed',
homepage: 'https://www.interconnects.ai',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Language model training and research, analysed',
enabledByDefault: true,
),
NewsSource(
id: 'ahead-of-ai',
name: 'Ahead of AI',
feedUrl: 'https://magazine.sebastianraschka.com/feed',
homepage: 'https://magazine.sebastianraschka.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'LLM research explained, by Sebastian Raschka',
enabledByDefault: true,
),
NewsSource(
id: 'lilianweng',
name: 'Lilian Weng',
feedUrl: 'https://lilianweng.github.io/index.xml',
homepage: 'https://lilianweng.github.io',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Thorough surveys of machine learning topics',
enabledByDefault: true,
),
NewsSource(
id: 'infoq',
name: 'InfoQ',
feedUrl: 'https://feed.infoq.com/',
homepage: 'https://www.infoq.com',
language: SourceLanguage.en,
kind: SourceKind.learning,
description: 'Software architecture and engineering talks, high volume',
),
// --- Fun ----------------------------------------------------------------
NewsSource(
id: 'xkcd',
name: 'xkcd',
feedUrl: 'https://xkcd.com/atom.xml',
homepage: 'https://xkcd.com',
language: SourceLanguage.en,
kind: SourceKind.fun,
description: 'Webcomic of romance, sarcasm, math and language',
enabledByDefault: true,
),
NewsSource(
id: 'thedailywtf',
name: 'The Daily WTF',
feedUrl: 'https://thedailywtf.com/rss',
homepage: 'https://thedailywtf.com',
language: SourceLanguage.en,
kind: SourceKind.fun,
description: 'IT horror stories, bad code and error messages',
enabledByDefault: true,
),
NewsSource(
id: 'programmerhumor',
name: 'r/ProgrammerHumor',
feedUrl: 'https://www.reddit.com/r/ProgrammerHumor/top/.rss?t=day',
homepage: 'https://www.reddit.com/r/ProgrammerHumor/',
language: SourceLanguage.en,
kind: SourceKind.fun,
description: "The day's top programming memes on Reddit",
),
];
/// The curated sources before the app remembered which ones it had seen.
+18
View File
@@ -8,6 +8,19 @@ enum SourceLanguage {
final String flag;
}
/// What a Source's articles are for; see GLOSSARY.md. Each kind has its own
/// section in the app and its own time window.
enum SourceKind {
news('News'),
learning('Learning'),
fun('Fun');
const SourceKind(this.label);
final String label;
}
/// A Source of any [kind]; the class name predates the kinds.
class NewsSource {
const NewsSource({
required this.id,
@@ -15,6 +28,7 @@ class NewsSource {
required this.feedUrl,
required this.homepage,
required this.language,
this.kind = SourceKind.news,
this.description = '',
this.enabledByDefault = false,
this.isCustom = false,
@@ -26,6 +40,7 @@ class NewsSource {
final String feedUrl;
final String homepage;
final SourceLanguage language;
final SourceKind kind;
final String description;
final bool enabledByDefault;
final bool isCustom;
@@ -40,6 +55,7 @@ class NewsSource {
'feedUrl': feedUrl,
'homepage': homepage,
'language': language.name,
'kind': kind.name,
'description': description,
};
@@ -49,6 +65,8 @@ class NewsSource {
feedUrl: json['feedUrl'] as String,
homepage: json['homepage'] as String? ?? '',
language: SourceLanguage.values.byName(json['language'] as String? ?? 'en'),
// Custom sources saved before kinds existed are News sources.
kind: SourceKind.values.asNameMap()[json['kind']] ?? SourceKind.news,
description: json['description'] as String? ?? '',
isCustom: true,
);
+29 -13
View File
@@ -49,9 +49,10 @@ class ArticleExtractor {
/// Hard cap on what we send to the model per article (~10k tokens).
static const maxChars = 40000;
/// Loads the article page and returns whichever is longer: the extracted
/// page or the full content some feeds ship. Never throws; on failure
/// (paywall, bot protection, CORS on web) it falls back to feed content.
/// Loads the article page and returns whichever has more text (on a tie,
/// more blocks): the extracted page or the full content some feeds ship.
/// Never throws; on failure (paywall, bot protection, CORS on web) it falls
/// back to feed content.
///
/// With [acceptConsentWalls], a redirect to a consent page the
/// [ConsentService] knows is accepted once and the article fetched again.
@@ -81,7 +82,12 @@ class ArticleExtractor {
if (page.wallUrl != null) return ArticleContent(fromFeed.blocks, consentWall: true);
if (page.status != 200) return fromFeed;
final fromPage = parse(FeedService.decodeBody(page.bytes), base);
return fromPage.plainText.length > fromFeed.plainText.length ? fromPage : fromFeed;
final pageText = fromPage.plainText.length, feedText = fromFeed.plainText.length;
// Without text either way (a comic), the one with the picture wins.
return pageText > feedText ||
(pageText == feedText && fromPage.blocks.length > fromFeed.blocks.length)
? fromPage
: fromFeed;
} catch (_) {
return fromFeed;
}
@@ -119,7 +125,8 @@ class ArticleExtractor {
/// Class/id fragments of page furniture that sits inside article bodies.
static final _junkMarker = RegExp(
r'comment|related|newsletter|share|social|promo|advert|paywall|subscribe|'
r'recommend|author-box|byline|breadcrumb|tags|toolbar|consent|cookie|card|native-ad',
r'recommend|author-box|byline|breadcrumb|tags|toolbar|consent|cookie|card|native-ad|'
r'license',
caseSensitive: false,
);
@@ -155,14 +162,15 @@ class ArticleExtractor {
final blocks = <ReaderBlock>[];
final seenImages = <String>{};
var chars = 0;
void add(BlockType type, String text) {
if (chars > maxChars) return;
bool add(BlockType type, String text) {
if (chars > maxChars) return false;
if (type == BlockType.image) {
if (!seenImages.add(text)) return;
if (!seenImages.add(text)) return false;
} else {
chars += text.length;
}
blocks.add(ReaderBlock(type, text));
return true;
}
void walk(dom.Element el) {
@@ -190,6 +198,8 @@ class ArticleExtractor {
case 'blockquote':
final t = _clean(el.text);
if (t.isNotEmpty) add(BlockType.quote, t);
// Quoted screenshots, as in The Daily WTF's Error'd.
el.querySelectorAll('img').forEach((img) => _addImage(img, base, add));
return;
case 'pre':
final t = el.text.trimRight();
@@ -221,7 +231,7 @@ class ArticleExtractor {
static String _clean(String s) => s.replaceAll(RegExp(r'\s+'), ' ').trim();
static void _addImage(
dom.Element img, Uri base, void Function(BlockType, String) add) {
dom.Element img, Uri base, bool Function(BlockType, String) add) {
// Lazy-loading sites keep the real URL in data attributes.
final raw = img.attributes['data-src'] ??
img.attributes['data-lazy-src'] ??
@@ -232,19 +242,25 @@ class ArticleExtractor {
}
final width = int.tryParse(img.attributes['width'] ?? '');
if (width != null && width < 120) return; // icons, avatars, tracking pixels
if (img.attributes.containsKey('usemap')) return; // clickable navigation
// Images linking to another page are author avatars or related-story
// thumbnails; links to the full-size image file are fine.
// thumbnails; links to the full-size image file or within the page are fine.
for (var p = img.parent; p != null; p = p.parent) {
if (p.localName == 'a') {
final href = p.attributes['href'] ?? '';
if (!RegExp(r'\.(jpe?g|png|webp|gif|avif)(\?|$)', caseSensitive: false)
.hasMatch(href)) {
if (!href.startsWith('#') &&
!RegExp(r'\.(jpe?g|png|webp|gif|avif)(\?|$)', caseSensitive: false).hasMatch(href)) {
return;
}
break;
}
}
final url = base.resolve(raw).toString();
if (url.startsWith('http')) add(BlockType.image, url);
if (!url.startsWith('http') || !add(BlockType.image, url)) return;
// Comics such as xkcd put a last joke in the title text.
final title = _clean(img.attributes['title'] ?? '');
if (title.length >= 20 && title != _clean(img.attributes['alt'] ?? '')) {
add(BlockType.caption, title);
}
}
}
+9 -5
View File
@@ -23,14 +23,15 @@ class FeedService {
static const _timeout = Duration(seconds: 20);
/// Fetches all [sources] in parallel. Failing feeds are reported in
/// Fetches all [sources] in parallel and drops articles older than the age
/// [maxAgeOf] gives for their Source. Failing feeds are reported in
/// [FeedResult.errors] instead of failing the whole refresh.
///
/// [corsProxy] is only used on web, where browsers block cross-origin feed
/// requests. It is prepended to the feed URL, e.g. `https://proxy/?url=`.
Future<FeedResult> fetchAll(
List<NewsSource> sources, {
required Duration maxAge,
required Duration Function(NewsSource source) maxAgeOf,
String corsProxy = '',
}) async {
final errors = <String, String>{};
@@ -43,12 +44,15 @@ class FeedService {
}
}));
final cutoff = DateTime.now().toUtc().subtract(maxAge);
final now = DateTime.now().toUtc();
final seen = <String>{};
final articles = <Article>[
for (final list in results)
for (final (i, list) in results.indexed)
for (final a in list)
if ((a.published == null || a.published!.isAfter(cutoff)) && seen.add(a.id)) a,
if ((a.published == null ||
a.published!.isAfter(now.subtract(maxAgeOf(sources[i])))) &&
seen.add(a.id))
a,
];
articles.sort((a, b) {
final pa = a.published, pb = b.published;
+15 -1
View File
@@ -9,6 +9,8 @@ import 'storage.dart';
/// Recognizes paid and sponsored articles so the feed can hide them.
///
/// Most paid articles are marked in the title (heise+, Golem's "(g+)").
/// Substack newsletters cut paid posts short in the feed and end them with a
/// "Read more" link to the post itself, so those are recognized from the feed.
/// SPIEGEL and t3n only mark them in the page's schema.org data
/// (`"isAccessibleForFree": false`), so for those sources each new article is
/// checked once in the background and the result is remembered.
@@ -33,6 +35,14 @@ class PaywallService {
caseSensitive: false,
);
/// What Substack appends to a paid post's feed content in place of the rest:
/// `<p><a href="{post url}">Read more</a></p>`, as the very end. Free posts
/// end with their own text, even when they ask readers to subscribe.
static final _substackCut = RegExp(
r'<p>\s*<a href="([^"]+)">\s*Read more\s*</a>\s*</p>\s*$',
caseSensitive: false,
);
static final _marker =
RegExp(r'''"isAccessibleForFree"\s*:\s*"?(true|false)"?''', caseSensitive: false);
@@ -41,7 +51,11 @@ class PaywallService {
static const _concurrency = 6;
static const _keepFor = Duration(days: 14);
bool isPaid(Article a) => paidTitle.hasMatch(a.title) || (_checks[a.link]?.paid ?? false);
bool isPaid(Article a) =>
paidTitle.hasMatch(a.title) || _isCutSubstackPost(a) || (_checks[a.link]?.paid ?? false);
static bool _isCutSubstackPost(Article a) =>
_substackCut.firstMatch(a.contentHtml)?.group(1) == a.link;
bool isSponsored(Article a) => sponsoredTitle.hasMatch(a.title);
+26 -7
View File
@@ -34,8 +34,10 @@ class AppState extends ChangeNotifier {
FeedService? feedService,
ArticleExtractor? extractor,
List<NewsSource> curatedSources = defaultSources,
Summarizer? summarizer,
}) : _feeds = feedService ?? FeedService(),
_curated = curatedSources,
_summarizerOverride = summarizer,
consent = ConsentService(),
paywall = PaywallService(_storage) {
_extractor = extractor ?? ArticleExtractor(consent: consent);
@@ -46,6 +48,9 @@ class AppState extends ChangeNotifier {
/// The curated catalog; [defaultSources] outside of tests.
final List<NewsSource> _curated;
/// Used instead of the AI set up in [settings]; for tests.
final Summarizer? _summarizerOverride;
late final ArticleExtractor _extractor;
final PaywallService paywall;
@@ -56,7 +61,7 @@ class AppState extends ChangeNotifier {
List<NewsSource> customSources = [];
Set<String> enabledSourceIds = {};
/// Everything the feeds returned; see [visibleArticles] for what is shown.
/// Everything the feeds returned; see [articlesOf] for what is shown.
List<Article> articles = [];
Map<String, String> feedErrors = {};
bool loading = false;
@@ -77,10 +82,22 @@ class AppState extends ChangeNotifier {
(settings.hidePaid && paywall.isPaid(a)) ||
(settings.hideSponsored && paywall.isSponsored(a));
/// Articles after the paid/sponsored filters. The feed and the briefing use these.
List<Article> get visibleArticles => articles.where((a) => !_isHidden(a)).toList();
/// Articles from Sources of [kind] after the paid/sponsored filters. Today
/// and the Briefing use News, the Learn tab the other kinds.
List<Article> articlesOf(SourceKind kind) =>
articles.where((a) => a.source.kind == kind && !_isHidden(a)).toList();
int get hiddenCount => articles.where(_isHidden).length;
/// Articles from Sources of [kind] that the paid/sponsored filters hide.
int hiddenCountOf(SourceKind kind) =>
articles.where((a) => a.source.kind == kind && _isHidden(a)).length;
/// How far back articles from Sources of [kind] reach. News follows the
/// user's setting; the other kinds age slower.
Duration windowOf(SourceKind kind) => switch (kind) {
SourceKind.news => Duration(hours: settings.maxAgeHours),
SourceKind.learning => const Duration(days: 14),
SourceKind.fun => const Duration(days: 3),
};
List<NewsSource> get allSources => [..._curated, ...customSources];
@@ -102,7 +119,7 @@ class AppState extends ChangeNotifier {
notifyListeners();
final result = await _feeds.fetchAll(
enabledSources,
maxAge: Duration(hours: settings.maxAgeHours),
maxAgeOf: (source) => windowOf(source.kind),
corsProxy: settings.corsProxy,
);
articles = result.articles;
@@ -151,6 +168,7 @@ class AppState extends ChangeNotifier {
required String name,
required String feedUrl,
required SourceLanguage language,
SourceKind kind = SourceKind.news,
}) async {
final uri = Uri.tryParse(feedUrl);
final source = NewsSource(
@@ -159,6 +177,7 @@ class AppState extends ChangeNotifier {
feedUrl: feedUrl,
homepage: uri == null ? '' : '${uri.scheme}://${uri.host}',
language: language,
kind: kind,
isCustom: true,
);
try {
@@ -229,7 +248,7 @@ class AppState extends ChangeNotifier {
// --- AI ------------------------------------------------------------------
Summarizer? get _summarizer => switch (settings.aiProvider) {
Summarizer? get _summarizer => _summarizerOverride ?? switch (settings.aiProvider) {
_ when !settings.aiConfigured => null,
AiProvider.claude => ClaudeSummarizer(
apiKey: settings.claudeApiKey, model: settings.claudeModel),
@@ -273,7 +292,7 @@ class AppState extends ChangeNotifier {
Future<void> generateBriefing() async {
final summarizer = _summarizer;
if (summarizer == null || briefingLoading) return;
final articles = visibleArticles;
final articles = articlesOf(SourceKind.news);
if (articles.isEmpty) {
briefingError = 'No articles loaded yet. Pull to refresh the feed first.';
notifyListeners();
+16 -5
View File
@@ -6,6 +6,7 @@ import 'package:flutter/material.dart';
import 'package:flutter_inappwebview/flutter_inappwebview.dart';
import '../models/article.dart';
import '../models/news_source.dart';
import '../services/article_extractor.dart';
import '../services/consent_service.dart';
import '../services/content_blocking.dart';
@@ -275,12 +276,22 @@ class _ReaderView extends StatelessWidget {
builder: (context, snapshot) {
final loaded = snapshot.data;
var blocks = loaded?.blocks ?? const <ReaderBlock>[];
// Comics and memes are the picture: show it whole in the body
// rather than as a cropped banner.
final pictureLed = article.source.kind == SourceKind.fun;
final banner = pictureLed ? null : article.imageUrl;
// The page's own hero image duplicates the feed image shown on top.
if (article.imageUrl != null) {
if (banner != null) {
blocks = blocks.skipWhile((b) => b.type == BlockType.image).toList();
}
if (pictureLed &&
article.imageUrl != null &&
!blocks.any((b) => b.type == BlockType.image)) {
blocks = [ReaderBlock(BlockType.image, article.imageUrl!), ...blocks];
}
final isFull = loaded != null &&
loaded.plainText.length >= max(400, article.excerpt.length * 2);
(loaded.plainText.length >= max(400, article.excerpt.length * 2) ||
(pictureLed && blocks.any((b) => b.type == BlockType.image)));
// Skip the lead if the body already starts with it.
final firstText = blocks.where((b) => b.type == BlockType.paragraph).firstOrNull;
final showLead = article.excerpt.isNotEmpty &&
@@ -291,9 +302,9 @@ class _ReaderView extends StatelessWidget {
return ListView(
padding: const EdgeInsets.only(bottom: 48),
children: [
if (article.imageUrl != null)
if (banner != null)
Image.network(
article.imageUrl!,
banner,
fit: BoxFit.cover,
webHtmlElementStrategy: WebHtmlElementStrategy.prefer,
// Take up space only once the image has loaded, so a broken
@@ -336,7 +347,7 @@ class _ReaderView extends StatelessWidget {
),
if (isFull)
for (final block in blocks)
if (block.text != article.imageUrl) _BlockView(block: block),
if (block.text != banner) _BlockView(block: block),
if (loaded != null) ...[
const SizedBox(height: 24),
const Divider(),
+3 -2
View File
@@ -1,6 +1,7 @@
import 'package:flutter/material.dart';
import '../models/briefing.dart';
import '../models/news_source.dart';
import '../state/app_state.dart';
import 'article_screen.dart';
import 'briefing_history_screen.dart';
@@ -59,7 +60,7 @@ class BriefingScreen extends StatelessWidget {
if (state.briefingLoading) {
return EmptyState(
icon: Icons.hourglass_top,
title: 'Reading ${state.visibleArticles.length} headlines…',
title: 'Reading ${state.articlesOf(SourceKind.news).length} headlines…',
message: 'This usually takes 10-60 seconds.',
action: const SizedBox(width: 160, child: LinearProgressIndicator()),
);
@@ -69,7 +70,7 @@ class BriefingScreen extends StatelessWidget {
icon: const Icon(Icons.auto_awesome),
label: Text(state.loading
? 'Waiting for feeds…'
: 'Generate from ${state.visibleArticles.length} articles'),
: 'Generate from ${state.articlesOf(SourceKind.news).length} articles'),
);
if (briefing == null) {
return EmptyState(
+2 -2
View File
@@ -25,13 +25,13 @@ class _FeedScreenState extends State<FeedScreen> {
listenable: widget.state,
builder: (context, _) {
final state = widget.state;
final all = state.visibleArticles;
final all = state.articlesOf(SourceKind.news);
final visible = all
.where((a) => _language == null || a.source.language == _language)
.where((a) => _sourceId == null || a.source.id == _sourceId)
.toList();
final sourcesInFeed = {for (final a in all) a.source.id: a.source};
final hidden = state.hiddenCount;
final hidden = state.hiddenCountOf(SourceKind.news);
return Scaffold(
appBar: AppBar(
+29 -14
View File
@@ -3,9 +3,13 @@ import 'package:flutter/material.dart';
import '../state/app_state.dart';
import 'briefing_screen.dart';
import 'feed_screen.dart';
import 'learn_screen.dart';
import 'settings_screen.dart';
import 'sources_screen.dart';
/// The tabs in the order they appear in the navigation bar.
enum _Tab { today, briefing, learn, sources, settings }
class HomeShell extends StatefulWidget {
const HomeShell({super.key, required this.state});
@@ -16,20 +20,20 @@ class HomeShell extends StatefulWidget {
}
class _HomeShellState extends State<HomeShell> {
int _index = 0;
_Tab _tab = _Tab.today;
bool _sourcesChanged = false;
void _select(int index) {
void _select(_Tab tab) {
// A text field left focused in Settings would otherwise get focus back
// (and open the keyboard) when returning from an article.
FocusManager.instance.primaryFocus?.unfocus();
// Source toggles only take effect on the next refresh; do it when the
// user leaves the Sources tab.
if (_index == 2 && index != 2 && _sourcesChanged) {
if (_tab == _Tab.sources && tab != _Tab.sources && _sourcesChanged) {
_sourcesChanged = false;
widget.state.refresh();
}
setState(() => _index = index);
setState(() => _tab = tab);
}
@override
@@ -37,20 +41,27 @@ class _HomeShellState extends State<HomeShell> {
final state = widget.state;
return Scaffold(
body: IndexedStack(
index: _index,
index: _tab.index,
children: [
FeedScreen(state: state),
BriefingScreen(state: state, openSettings: () => _select(3)),
SourcesScreen(state: state, onChanged: () => _sourcesChanged = true),
SettingsScreen(state: state),
].indexed
for (final tab in _Tab.values)
// Hidden tabs stay alive but must not hold or regain focus.
.map((e) => ExcludeFocus(excluding: e.$1 != _index, child: e.$2))
.toList(),
ExcludeFocus(
excluding: tab != _tab,
child: switch (tab) {
_Tab.today => FeedScreen(state: state),
_Tab.briefing => BriefingScreen(
state: state, openSettings: () => _select(_Tab.settings)),
_Tab.learn => LearnScreen(state: state),
_Tab.sources =>
SourcesScreen(state: state, onChanged: () => _sourcesChanged = true),
_Tab.settings => SettingsScreen(state: state),
},
),
],
),
bottomNavigationBar: NavigationBar(
selectedIndex: _index,
onDestinationSelected: _select,
selectedIndex: _tab.index,
onDestinationSelected: (i) => _select(_Tab.values[i]),
destinations: const [
NavigationDestination(
icon: Icon(Icons.newspaper_outlined),
@@ -60,6 +71,10 @@ class _HomeShellState extends State<HomeShell> {
icon: Icon(Icons.auto_awesome_outlined),
selectedIcon: Icon(Icons.auto_awesome),
label: 'Briefing'),
NavigationDestination(
icon: Icon(Icons.school_outlined),
selectedIcon: Icon(Icons.school),
label: 'Learn'),
NavigationDestination(
icon: Icon(Icons.rss_feed_outlined),
selectedIcon: Icon(Icons.rss_feed),
+98
View File
@@ -0,0 +1,98 @@
import 'package:flutter/material.dart';
import '../models/news_source.dart';
import '../state/app_state.dart';
import 'article_screen.dart';
import 'feed_screen.dart';
/// Reading that keeps for days, from the Sources that aren't News: one
/// section per kind, each with its own time window.
class LearnScreen extends StatelessWidget {
const LearnScreen({super.key, required this.state});
final AppState state;
/// The sections, top to bottom.
static const kinds = [SourceKind.learning, SourceKind.fun];
@override
Widget build(BuildContext context) {
return ListenableBuilder(
listenable: state,
builder: (context, _) => Scaffold(
appBar: AppBar(
title: const Text('Learn'),
actions: [
IconButton(
tooltip: 'Refresh',
onPressed: state.loading ? null : state.refresh,
icon: const Icon(Icons.refresh),
),
],
),
body: Column(
children: [
if (state.loading) const LinearProgressIndicator(),
Expanded(
child: RefreshIndicator(
onRefresh: state.refresh,
child: ListView(
padding: const EdgeInsets.fromLTRB(12, 0, 12, 24),
children: [for (final kind in kinds) ..._section(context, kind)],
),
),
),
],
),
),
);
}
List<Widget> _section(BuildContext context, SourceKind kind) {
final theme = Theme.of(context);
final muted = theme.colorScheme.onSurfaceVariant;
final articles = state.articlesOf(kind);
final hidden = state.hiddenCountOf(kind);
final hasSources = state.enabledSources.any((s) => s.kind == kind);
return [
Padding(
padding: const EdgeInsets.fromLTRB(4, 16, 4, 8),
child: Column(
crossAxisAlignment: CrossAxisAlignment.start,
children: [
Text(kind.label,
style: theme.textTheme.titleMedium?.copyWith(color: theme.colorScheme.primary)),
Text(
'${articles.length} articles · last ${_describe(state.windowOf(kind))}'
'${hidden > 0 ? ' · $hidden paid/ads hidden' : ''}',
style: theme.textTheme.bodySmall?.copyWith(color: muted),
),
],
),
),
if (articles.isEmpty && !state.loading)
Padding(
padding: const EdgeInsets.fromLTRB(4, 8, 4, 16),
child: Text(
hasSources
? 'Nothing new in this window. Pull down to refresh.'
: 'No ${kind.label} sources enabled. Turn some on in the Sources tab.',
style: theme.textTheme.bodyMedium?.copyWith(color: muted),
),
),
for (final article in articles)
Padding(
padding: const EdgeInsets.only(bottom: 8),
child: ArticleCard(
article: article,
hasSummary: state.summaries[article.id]?.text != null,
isPaid: state.paywall.isPaid(article),
onTap: () => openArticle(context, state, article),
),
),
];
}
static String _describe(Duration window) =>
window.inHours < 72 ? '${window.inHours} h' : '${window.inDays} days';
}
+50 -24
View File
@@ -40,29 +40,7 @@ class SourcesScreen extends StatelessWidget {
body: ListView(
padding: const EdgeInsets.only(bottom: 96),
children: [
for (final lang in SourceLanguage.values) ...[
Padding(
padding: const EdgeInsets.fromLTRB(16, 16, 16, 4),
child: Text('${lang.flag} ${lang.label}',
style: theme.textTheme.titleSmall
?.copyWith(color: theme.colorScheme.primary)),
),
for (final source in state.allSources.where((s) => s.language == lang))
_SourceTile(
source: source,
enabled: state.enabledSourceIds.contains(source.id),
onToggle: (v) {
state.setSourceEnabled(source, v);
onChanged();
},
onDelete: source.isCustom
? () {
state.removeCustomSource(source);
onChanged();
}
: null,
),
],
for (final kind in SourceKind.values) ..._kindSection(context, kind),
],
),
);
@@ -70,6 +48,44 @@ class SourcesScreen extends StatelessWidget {
);
}
/// One section per kind, split by language. Kinds and languages without
/// Sources are left out instead of showing an empty header.
List<Widget> _kindSection(BuildContext context, SourceKind kind) {
final theme = Theme.of(context);
final sources = state.allSources.where((s) => s.kind == kind).toList();
if (sources.isEmpty) return const [];
return [
Padding(
padding: const EdgeInsets.fromLTRB(16, 24, 16, 0),
child: Text(kind.label, style: theme.textTheme.titleMedium),
),
for (final lang in SourceLanguage.values)
if (sources.any((s) => s.language == lang)) ...[
Padding(
padding: const EdgeInsets.fromLTRB(16, 12, 16, 4),
child: Text('${lang.flag} ${lang.label}',
style: theme.textTheme.titleSmall
?.copyWith(color: theme.colorScheme.primary)),
),
for (final source in sources.where((s) => s.language == lang))
_SourceTile(
source: source,
enabled: state.enabledSourceIds.contains(source.id),
onToggle: (v) {
state.setSourceEnabled(source, v);
onChanged();
},
onDelete: source.isCustom
? () {
state.removeCustomSource(source);
onChanged();
}
: null,
),
],
];
}
Future<void> _addFeed(BuildContext context) async {
final added = await showDialog<bool>(
context: context,
@@ -134,6 +150,7 @@ class _AddFeedDialogState extends State<_AddFeedDialog> {
final _name = TextEditingController();
final _url = TextEditingController();
SourceLanguage _language = SourceLanguage.de;
SourceKind _kind = SourceKind.news;
bool _busy = false;
String? _error;
@@ -156,7 +173,7 @@ class _AddFeedDialogState extends State<_AddFeedDialog> {
_error = null;
});
final error = await widget.state
.addCustomSource(name: name, feedUrl: url, language: _language);
.addCustomSource(name: name, feedUrl: url, language: _language, kind: _kind);
if (!mounted) return;
if (error == null) {
Navigator.pop(context, true);
@@ -199,6 +216,15 @@ class _AddFeedDialogState extends State<_AddFeedDialog> {
selected: {_language},
onSelectionChanged: (s) => setState(() => _language = s.first),
),
const SizedBox(height: 12),
SegmentedButton<SourceKind>(
segments: [
for (final k in SourceKind.values)
ButtonSegment(value: k, label: Text(k.label)),
],
selected: {_kind},
onSelectionChanged: (s) => setState(() => _kind = s.first),
),
if (_error != null) ...[
const SizedBox(height: 12),
Text(_error!, style: TextStyle(color: Theme.of(context).colorScheme.error)),
+78
View File
@@ -102,8 +102,86 @@ void main() {
isFragment: true);
expect(content.blocks, hasLength(2));
});
test('an xkcd page gives the comic and its title text, without the page around it', () {
final content = ArticleExtractor.parse(xkcdPage, Uri.parse('https://xkcd.com/3312/'));
expect(content.blocks.map((b) => (b.type, b.text)), [
(BlockType.image, 'https://imgs.xkcd.com/comics/momentum_demonstration.png'),
(BlockType.caption, xkcdTitleText),
]);
});
test("The Daily WTF's Error'd screenshots show between their stories", () {
final content = ArticleExtractor.parse(
errordPage, Uri.parse('https://thedailywtf.com/articles/scrollpocalypse'));
expect(content.blocks.map((b) => b.type), [
BlockType.paragraph,
BlockType.image,
BlockType.paragraph,
BlockType.image,
]);
expect(content.blocks[1].text, 'https://cdn.example.com/errord/first.png');
});
}
const xkcdTitleText = '"I\'m doing really well on the Hertzsprung-Russell academic diagram, '
'which has spectral class on one axis and Rate My Professors rating on the other."';
/// Trimmed from https://xkcd.com/3312/.
const xkcdPage = '''
<!DOCTYPE html>
<html><head><title>xkcd: Momentum Demonstration</title></head>
<body>
<div id="topContainer">
<div id="topLeft"><ul><li><a href="/archive">Archive</a></li><li><a href="/about">About</a></li></ul></div>
<div id="topRight">
<div id="masthead"><span><a href="/"><img src="/s/0b7742.png" alt="xkcd.com logo" height="83" width="185"/></a></span>
<span id="slogan">A webcomic of romance,<br/> sarcasm, math, and language.</span></div>
<div id="news"><div id="xkcdNews"><a href="https://store.xkcd.com/"><img src="https://imgs.xkcd.com/store_news/news_back.png" alt="The xkcd store is back, with shirts, stickers, and posters for sale." title="xkcd store news" /></a></div></div>
</div>
</div>
<div id="middleContainer" class="box">
<div id="ctitle">Momentum Demonstration</div>
<ul class="comicNav"><li><a href="/1/">|&lt;</a></li><li><a rel="prev" href="/3311/">&lt; Prev</a></li></ul>
<div id="comic">
<img src="//imgs.xkcd.com/comics/momentum_demonstration.png" title="&quot;I&#39;m doing really well on the Hertzsprung-Russell academic diagram, which has spectral class on one axis and Rate My Professors rating on the other.&quot;" alt="Momentum Demonstration" srcset="//imgs.xkcd.com/comics/momentum_demonstration_2x.png 2x" style="image-orientation:none" />
</div>
<ul class="comicNav"><li><a href="/1/">|&lt;</a></li><li><a rel="prev" href="/3311/">&lt; Prev</a></li></ul>
<br />
Permanent link to this comic: <a href="https://xkcd.com/3312">https://xkcd.com/3312/</a><br />
Image URL (for hotlinking/embedding): <a href= "https://imgs.xkcd.com/comics/momentum_demonstration.png">https://imgs.xkcd.com/comics/momentum_demonstration.png</a>
<div id="transcript" style="display: none"></div>
</div>
<div id="bottom" class="box">
<img src="//imgs.xkcd.com/s/a899e84.jpg" width="520" height="100" alt="Selected Comics" usemap="#comicmap"/>
<map id="comicmap" name="comicmap"><area shape="rect" coords="0,0,100,100" href="/150/" alt="Grownups"/></map>
<div id="comicLinks">Comics I enjoy:<br/><a href="http://threewordphrase.com/">Three Word Phrase</a></div>
<div id="footnote" style="width:70%">xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3&plusmn;1 emulated in Javascript on an Apple IIGS</div>
<div id="licenseText">
<p>
This work is licensed under a
<a href="https://creativecommons.org/licenses/by-nc/2.5/">Creative Commons Attribution-NonCommercial 2.5 License</a>.
</p><p>
This means you're free to copy and share these comics (but not to sell them). <a rel="license" href="/license.html">More details</a>.</p>
</div>
</div>
</body></html>''';
/// Trimmed from https://thedailywtf.com/articles/scrollpocalypse.
const errordPage = '''
<html><body>
<div itemscope itemtype="http://schema.org/Article" class="article-container errord">
<div itemprop="articleBody" class="article-body">
<p>Dr. Doom...scroller titled this one "Scrollbar-ception". Said he "Screenshot of the TikTok webpage."</p>
<blockquote><p><a href="#d6711f9e"><img itemprop="image" border="0" alt="d6711f9e" src="https://cdn.example.com/errord/first.png"/></a></p><p> </p></blockquote>
<p>An anonymous reader clarified "I was reorganizing the document by drag-and-dropping table cells."</p>
<blockquote><p><a href="#01270f37"><img itemprop="image" border="0" alt="01270f37" src="https://cdn.example.com/errord/second.png"/></a></p><p> </p></blockquote>
</div>
</div>
</body></html>''';
/// Answers every request as if the server had redirected to [finalUrl].
class _RedirectingClient extends http.BaseClient {
_RedirectingClient(this.finalUrl, this.body);
+109
View File
@@ -0,0 +1,109 @@
import 'dart:io';
import 'dart:ui' as ui;
import 'package:daily_tech_news/models/article.dart';
import 'package:daily_tech_news/models/news_source.dart';
import 'package:daily_tech_news/services/article_extractor.dart';
import 'package:daily_tech_news/ui/article_screen.dart';
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:http/http.dart' as http;
import 'package:http/testing.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'support/fake_app.dart';
/// A plain PNG of the given size, standing in for a comic strip or meme.
Future<List<int>> _png(int width, int height) async {
final recorder = ui.PictureRecorder();
Canvas(recorder).drawRect(
Rect.fromLTWH(0, 0, width.toDouble(), height.toDouble()), Paint()..color = Colors.black);
final image = await recorder.endRecording().toImage(width, height);
return (await image.toByteData(format: ui.ImageByteFormat.png))!.buffer.asUint8List();
}
/// Serves a [width] x [height] PNG from this machine, so the reader view can
/// load and lay out a real picture. Returns its URL.
Future<String> _serveImage(WidgetTester tester, int width, int height) async {
HttpOverrides.global = null;
final server = (await tester.runAsync(() => HttpServer.bind(InternetAddress.loopbackIPv4, 0)))!;
addTearDown(() => tester.runAsync(server.close));
final png = (await tester.runAsync(() => _png(width, height)))!;
server.listen((request) => request.response
// No idle keep-alive timer left behind in the test's fake time.
..persistentConnection = false
..headers.contentType = ContentType('image', 'png')
..add(png)
..close());
return 'http://127.0.0.1:${server.port}/picture.png';
}
/// Opens [article] in the reader, with the article page answered by [page],
/// and waits until the picture at [imageUrl] has loaded. Returns its finder.
Future<Finder> _openAndLoad(
WidgetTester tester, Article article, String imageUrl, http.Response page) async {
final state = (await tester.runAsync(() => launchApp([article.source],
extractor: ArticleExtractor(client: MockClient((_) async => page)))))!;
await tester.pumpWidget(MaterialApp(home: ArticleScreen(state: state, article: article)));
final picture = find.descendant(
of: find.byWidgetPredicate((w) =>
w is Image && w.image is NetworkImage && (w.image as NetworkImage).url == imageUrl),
matching: find.byType(RawImage));
for (var i = 0; i < 100; i++) {
await tester.runAsync(() => Future<void>.delayed(const Duration(milliseconds: 20)));
await tester.pump();
if (picture.evaluate().isNotEmpty && tester.widget<RawImage>(picture).image != null) break;
}
return picture;
}
double _aspect(WidgetTester tester, Finder picture) {
final size = tester.getSize(picture);
return size.height / size.width;
}
const _titleText = 'The title text is the last joke, hidden in the mouse-over.';
void main() {
setUp(() => SharedPreferences.setMockInitialValues({}));
testWidgets('opening an xkcd comic shows the whole comic and its title text', (tester) async {
final comicUrl = await _serveImage(tester, 300, 100);
final page = '''
<html><body>
<div id="ctitle">Strip</div>
<div id="comic"><img src="$comicUrl" title="$_titleText" alt="Strip"></div>
<div id="licenseText"><p>This work is licensed under a Creative Commons Attribution-NonCommercial 2.5 License.</p></div>
</body></html>''';
// What the xkcd feed gives: a title, a link and the comic as the image.
final article = Article(
title: 'Strip',
link: 'https://xkcd.com/9999/',
source: testSource('xkcd', SourceKind.fun),
imageUrl: comicUrl);
final comic = await _openAndLoad(tester, article, comicUrl,
http.Response(page, 200, headers: {'content-type': 'text/html; charset=utf-8'}));
expect(tester.widget<RawImage>(comic).image, isNotNull, reason: 'the comic loads');
expect(_aspect(tester, comic), closeTo(100 / 300, 0.01), reason: 'the comic is not cropped');
expect(find.text(_titleText), findsOneWidget);
expect(find.textContaining("couldn't be loaded"), findsNothing);
});
testWidgets('a meme whose page blocks the reader still shows the picture from the feed',
(tester) async {
final memeUrl = await _serveImage(tester, 100, 200);
final article = Article(
title: 'vibeCoding',
link: 'https://www.reddit.com/r/ProgrammerHumor/comments/1/vibecoding/',
source: testSource('programmerhumor', SourceKind.fun),
imageUrl: memeUrl);
final meme = await _openAndLoad(tester, article, memeUrl, http.Response('Blocked', 403));
expect(tester.widget<RawImage>(meme).image, isNotNull, reason: 'the meme loads');
expect(_aspect(tester, meme), closeTo(200 / 100, 0.01), reason: 'the meme is not cropped');
expect(find.textContaining("couldn't be loaded"), findsNothing);
});
}
+35 -1
View File
@@ -8,7 +8,7 @@ void main() {
test('the News sources on by default are the curated quality set', () {
final onByDefault = {
for (final s in defaultSources)
if (s.enabledByDefault) s.id,
if (s.enabledByDefault && s.kind == SourceKind.news) s.id,
};
expect(onByDefault, {
@@ -43,6 +43,40 @@ void main() {
expect(ids, isNot(contains('engadget')));
});
test('the Learning sources are the curated set, all English, with the agreed defaults', () {
final learning = {
for (final s in defaultSources)
if (s.kind == SourceKind.learning) s.name: (s.feedUrl, s.language, s.enabledByDefault),
};
const en = SourceLanguage.en;
expect(learning, {
'ByteByteGo': ('https://blog.bytebytego.com/feed', en, true),
'System Design One': ('https://newsletter.systemdesign.one/feed', en, true),
'The Pragmatic Engineer (blog)': ('https://blog.pragmaticengineer.com/rss/', en, true),
'Martin Fowler': ('https://martinfowler.com/feed.atom', en, true),
'Julia Evans': ('https://jvns.ca/atom.xml', en, true),
'Dan Luu': ('https://danluu.com/atom.xml', en, true),
'The Cloudflare Blog': ('https://blog.cloudflare.com/rss/', en, true),
'Netflix Technology Blog': ('https://netflixtechblog.com/feed', en, true),
'Engineering at Meta': ('https://engineering.fb.com/feed/', en, true),
'GitHub Blog: Engineering': ('https://github.blog/engineering/feed/', en, true),
'Latent Space': ('https://www.latent.space/feed', en, true),
'Import AI': ('https://importai.substack.com/feed', en, true),
'Interconnects': ('https://www.interconnects.ai/feed', en, true),
'Ahead of AI': ('https://magazine.sebastianraschka.com/feed', en, true),
'Lilian Weng': ('https://lilianweng.github.io/index.xml', en, true),
'InfoQ': ('https://feed.infoq.com/', en, false),
});
});
test('every Learning source says what it is for and links its homepage', () {
for (final s in defaultSources.where((s) => s.kind == SourceKind.learning)) {
expect(s.description, isNotEmpty, reason: s.name);
expect(Uri.parse(s.homepage).isScheme('https'), isTrue, reason: s.name);
}
});
test('every Source has its own id', () {
final ids = defaultSources.map((s) => s.id).toList();
+15
View File
@@ -0,0 +1,15 @@
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Ahead of AI]]></title><link>https://magazine.sebastianraschka.com</link><generator>Substack</generator>
<!-- Real items from https://magazine.sebastianraschka.com/feed (October 2026), content:encoded trimmed to its first paragraph and its end. -->
<!-- Paid post (audience "only_paid" on its page): Substack cuts the content and ends it with a "Read more" link to the post. -->
<item><title><![CDATA[Building an AI Text Detector From Scratch]]></title><description><![CDATA[An End-to-End Project With Dataset Construction, Model Training, Local Deployment, and RLVR]]></description><link>https://magazine.sebastianraschka.com/p/ai-detector-from-scratch</link><guid isPermaLink="false">https://magazine.sebastianraschka.com/p/ai-detector-from-scratch</guid><dc:creator><![CDATA[Sebastian Raschka, PhD]]></dc:creator><pubDate>Sat, 15 Aug 2026 11:54:24 GMT</pubDate><content:encoded><![CDATA[<p>Substack recently launched its AI detector feature in the UI, which is super interesting.</p><p>For this, we are going to fine-tune a DistilBERT classifier (similar to what I described in one of my early Substack articles, <a href="https://magazine.sebastianraschka.com/p/finetuning-large-language-models">Finetuning Large Language Models</a>), but more details on that later when we get to that stage.</p><p></p>
<p>
<a href="https://magazine.sebastianraschka.com/p/ai-detector-from-scratch">
Read more
</a>
</p>
]]></content:encoded></item>
<!-- Free post (audience "everyone") whose footer asks readers to become paid subscribers. -->
<item><title><![CDATA[Controlling Reasoning Effort in LLMs]]></title><description><![CDATA[How LLMs Learn Low-, Medium-, and High-Effort Reasoning Modes]]></description><link>https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-in-llms</link><guid isPermaLink="false">https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-in-llms</guid><dc:creator><![CDATA[Sebastian Raschka, PhD]]></dc:creator><pubDate>Sat, 18 Jul 2026 11:16:09 GMT</pubDate><content:encoded><![CDATA[<p><span>It has been almost two years since OpenAI released o1, a model that popularized the idea of LLM-based reasoning models. DeepSeek-R1 followed about four months later, together with details of a reinforcement learning with verifiable rewards (RLVR) recipe to train such reasoning models.</span></p><p>If you liked my previous <a href="https://amzn.to/4fqvn0D">Build a Large Language Model (From Scratch)</a> book, this is essentially a sequel implementing inference-time scaling techniques and reinforcement learning algorithms from scratch.</p><p>And if you want to support future long-form articles like this one, consider <a href="https://magazine.sebastianraschka.com/subscribe">becoming a paid subscriber</a>. It helps me keep writing these independent deep dives and sharing the accompanying code, figures, and experiments.</p><p></p>]]></content:encoded></item>
<!-- Free post (audience "everyone") that ends with Substack's subscribe widget. -->
<item><title><![CDATA[GPT-6 Astra, Looped Transformers, and Hidden Reasoning]]></title><description><![CDATA[A Look at Recurrent Depth, Hidden Chains of Thought, and Recent Research on Looping Transformer Blocks]]></description><link>https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and</link><guid isPermaLink="false">https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and</guid><dc:creator><![CDATA[Sebastian Raschka, PhD]]></dc:creator><pubDate>Wed, 09 Sep 2026 11:14:25 GMT</pubDate><content:encoded><![CDATA[<p>A lot has happened in the last few weeks. I am sure that OpenAI&#8217;s GPT-6 Astra is top of mind for everyone right now. In particular, thoughts on its performance, the looped transformer/recurrent depth aspects, and rumors that Astra is &#8220;hiding&#8221; its reasoning trace (i.e., chain of thought).</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://magazine.sebastianraschka.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item>
</channel></rss>
+13
View File
@@ -0,0 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Interconnects AI]]></title><link>https://www.interconnects.ai</link><generator>Substack</generator>
<!-- Real items from https://www.interconnects.ai/feed (October 2026), content:encoded trimmed to its first paragraph and its end. -->
<!-- Paid post (audience "only_paid" on its page). -->
<item><title><![CDATA[Lessons from the hacks]]></title><description><![CDATA[Musings on model alignment, what determines safety, and where we go from here.]]></description><link>https://www.interconnects.ai/p/lessons-from-the-hacks</link><guid isPermaLink="false">https://www.interconnects.ai/p/lessons-from-the-hacks</guid><dc:creator><![CDATA[Nathan Lambert]]></dc:creator><pubDate>Sun, 09 Aug 2026 14:57:11 GMT</pubDate><content:encoded><![CDATA[<p><span>The recent run of cyberattacks by in-development frontier models has got me thinking a lot about how our current incentive systems are not well suited for such fast technological transitions.</span></p><div><hr></div><h4>Book sale</h4><p><span>I&#8217;m also hosting a book launch where you can get a free signed copy tomorrow from 5-8PM in Seattle (Fremont/Ballard area) &#8211; we still have some extra space so I&#8217;m opening signups to paid subscribers below the paywall:</span></p>
<p>
<a href="https://www.interconnects.ai/p/lessons-from-the-hacks">
Read more
</a>
</p>
]]></content:encoded></item>
<!-- Free post (audience "everyone") that mentions paid subscribers and ends with an embedded post's "Read more". -->
<item><title><![CDATA[When will average people feel AI’s impact?]]></title><description><![CDATA[We&#8217;re <5 years into a compounding revolution which could take a century, and how the AI industry should manage this.]]></description><link>https://www.interconnects.ai/p/when-will-average-people-feel-ais</link><guid isPermaLink="false">https://www.interconnects.ai/p/when-will-average-people-feel-ais</guid><dc:creator><![CDATA[Nathan Lambert]]></dc:creator><pubDate>Wed, 09 Sep 2026 11:01:23 GMT</pubDate><content:encoded><![CDATA[<h5>Housekeeping: Paid subscribers to Interconnects now get a permanent 40% discount on my book when purchasing at Manning.com. Access the code at the Interconnects <a href="https://www.interconnects.ai/perks">perks page</a>.</h5><p>Many AI optimists tend to compare what is happening in this AI boom to the industrial revolution, or to other periods of rapid technological advancement and diffusion into society.</p><div><a class="embedded-post" native="true" href="https://www.fabricatedknowledge.com/p/mythos-and-engels-pause?utm_source=substack&amp;utm_campaign=post_embed&amp;utm_medium=web&amp;embedding_publication_id=48206"><div class="embedded-post-header"><span class="embedded-post-publication-name">Fabricated Knowledge</span></div><div class="embedded-post-title-wrapper"><div class="embedded-post-title">Engels' Pause and the Permanent Underclass </div></div><div class="embedded-post-cta-wrapper"><span class="embedded-post-cta">Read more</span></div><div class="embedded-post-meta">6 months ago &#183; 214 likes &#183; 5 comments &#183; Doug O'Laughlin</div></a></div>]]></content:encoded></item>
</channel></rss>
+102
View File
@@ -0,0 +1,102 @@
import 'package:daily_tech_news/data/default_sources.dart';
import 'package:daily_tech_news/models/article.dart';
import 'package:daily_tech_news/models/briefing.dart';
import 'package:daily_tech_news/models/news_source.dart';
import 'package:daily_tech_news/services/summarizer.dart';
import 'package:daily_tech_news/state/app_state.dart';
import 'package:daily_tech_news/ui/feed_screen.dart';
import 'package:daily_tech_news/ui/learn_screen.dart';
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'support/fake_app.dart';
/// Remembers which articles a Briefing was asked for.
class _RecordingSummarizer implements Summarizer {
List<Article>? briefingInput;
@override
String get name => 'test';
@override
Future<String> summarizeArticle(Article article, String text, SourceLanguage language) async =>
'';
@override
Future<DailyBriefing> createBriefing(List<Article> articles, SourceLanguage language) async {
briefingInput = articles;
return DailyBriefing(createdAt: DateTime.now(), intro: '', topics: const [], generatedBy: name);
}
}
void main() {
final verge = testSource('verge', SourceKind.news);
final bytebytego = testSource('bytebytego', SourceKind.learning);
final xkcd = testSource('xkcd', SourceKind.fun);
final feeds = {
verge.feedUrl: [('News from this morning', const Duration(hours: 2))],
bytebytego.feedUrl: [('Explainer from 2 days ago', const Duration(days: 2))],
xkcd.feedUrl: [('Comic from today', const Duration(hours: 5)),
('Comic from 2 days ago', const Duration(days: 2)),
('Comic from 4 days ago', const Duration(days: 4))],
};
setUp(() => SharedPreferences.setMockInitialValues({}));
Future<AppState> launch(WidgetTester tester, {Summarizer? summarizer}) async =>
(await tester.runAsync(() => launchApp([verge, bytebytego, xkcd],
feeds: feeds, summarizer: summarizer)))!;
double top(WidgetTester tester, String text) => tester.getTopLeft(find.text(text)).dy;
test('the curated Fun sources are xkcd and The Daily WTF on, r/ProgrammerHumor off', () {
final fun = {
for (final s in defaultSources)
if (s.kind == SourceKind.fun) s.name: (s.feedUrl, s.language, s.enabledByDefault),
};
expect(fun, {
'xkcd': ('https://xkcd.com/atom.xml', SourceLanguage.en, true),
'The Daily WTF': ('https://thedailywtf.com/rss', SourceLanguage.en, true),
'r/ProgrammerHumor': (
'https://www.reddit.com/r/ProgrammerHumor/top/.rss?t=day',
SourceLanguage.en,
false,
),
});
});
testWidgets('the Learn tab has a Fun section below Learning with the last 3 days of Fun',
(tester) async {
final state = await launch(tester);
await tester.pumpWidget(MaterialApp(home: LearnScreen(state: state)));
expect(find.text('Fun'), findsOneWidget);
expect(find.text('2 articles · last 3 days'), findsOneWidget);
expect(top(tester, 'Learning'), lessThan(top(tester, 'Fun')));
expect(top(tester, 'Explainer from 2 days ago'), lessThan(top(tester, 'Fun')));
expect(top(tester, 'Fun'), lessThan(top(tester, 'Comic from today')));
expect(top(tester, 'Comic from today'), lessThan(top(tester, 'Comic from 2 days ago')));
expect(find.text('Comic from 4 days ago'), findsNothing);
});
testWidgets('Today leaves out Fun articles', (tester) async {
final state = await launch(tester);
await tester.pumpWidget(MaterialApp(home: FeedScreen(state: state)));
expect(find.text('News from this morning'), findsOneWidget);
expect(find.text('Comic from today'), findsNothing);
});
testWidgets('the Briefing leaves out Fun articles', (tester) async {
final summarizer = _RecordingSummarizer();
final state = await launch(tester, summarizer: summarizer);
await tester.runAsync(state.generateBriefing);
expect(summarizer.briefingInput!.map((a) => a.title), ['News from this morning']);
});
}
+92
View File
@@ -0,0 +1,92 @@
import 'package:daily_tech_news/models/news_source.dart';
import 'package:daily_tech_news/state/app_state.dart';
import 'package:daily_tech_news/ui/feed_screen.dart';
import 'package:daily_tech_news/ui/home_shell.dart';
import 'package:daily_tech_news/ui/learn_screen.dart';
import 'package:daily_tech_news/ui/settings_screen.dart';
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'support/fake_app.dart';
void main() {
final verge = testSource('verge', SourceKind.news);
final bytebytego = testSource('bytebytego', SourceKind.learning);
final feeds = {
verge.feedUrl: [('News from this morning', const Duration(hours: 2))],
bytebytego.feedUrl: [('Explainer from 13 days ago', const Duration(days: 13)),
('Explainer from 2 days ago', const Duration(days: 2))],
};
setUp(() => SharedPreferences.setMockInitialValues({}));
Future<AppState> launch(WidgetTester tester, [FakeFeeds? feedService]) async =>
(await tester.runAsync(() => launchApp([verge, bytebytego],
feedService: feedService ?? FakeFeeds(feeds))))!;
Finder tab(String label) =>
find.descendant(of: find.byType(NavigationBar), matching: find.text(label));
testWidgets('the Learn tab lists Learning articles newest first, without News',
(tester) async {
final state = await launch(tester);
await tester.pumpWidget(MaterialApp(home: LearnScreen(state: state)));
final newer = find.text('Explainer from 2 days ago');
final older = find.text('Explainer from 13 days ago');
expect(newer, findsOneWidget);
expect(older, findsOneWidget);
expect(tester.getTopLeft(newer).dy, lessThan(tester.getTopLeft(older).dy));
expect(find.text('News from this morning'), findsNothing);
});
testWidgets('Today lists News without Learning articles', (tester) async {
final state = await launch(tester);
await tester.pumpWidget(MaterialApp(home: FeedScreen(state: state)));
expect(find.text('News from this morning'), findsOneWidget);
expect(find.text('Explainer from 2 days ago'), findsNothing);
});
testWidgets('the tabs are Today, Briefing, Learn, Sources and Settings', (tester) async {
final state = await launch(tester);
await tester.pumpWidget(MaterialApp(home: HomeShell(state: state)));
final labels = ['Today', 'Briefing', 'Learn', 'Sources', 'Settings'];
final xs = [for (final label in labels) tester.getCenter(tab(label)).dx];
expect(xs, [...xs]..sort());
});
testWidgets('Open Settings on the Briefing tab goes to Settings', (tester) async {
final state = await launch(tester);
await tester.pumpWidget(MaterialApp(home: HomeShell(state: state)));
await tester.tap(tab('Briefing'));
await tester.pump();
await tester.tap(find.text('Open Settings'));
await tester.pump();
expect(find.byType(SettingsScreen), findsOneWidget);
});
testWidgets('leaving the Sources tab after a change refreshes the feeds', (tester) async {
final feedService = FakeFeeds(feeds);
final state = await launch(tester, feedService);
await tester.pumpWidget(MaterialApp(home: HomeShell(state: state)));
final before = feedService.refreshes;
await tester.tap(tab('Sources'));
await tester.pump();
await tester.tap(find.text('verge'));
await tester.pump();
expect(feedService.refreshes, before);
await tester.tap(tab('Learn'));
await tester.pump();
expect(feedService.refreshes, before + 1);
});
}
+17
View File
@@ -2,10 +2,14 @@
import 'dart:io';
import 'package:daily_tech_news/data/default_sources.dart';
import 'package:daily_tech_news/models/news_source.dart';
import 'package:daily_tech_news/services/article_extractor.dart';
import 'package:daily_tech_news/services/consent_service.dart';
import 'package:daily_tech_news/services/feed_service.dart';
import 'package:daily_tech_news/services/storage.dart';
import 'package:daily_tech_news/state/app_state.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:shared_preferences/shared_preferences.dart';
void main() {
final live = Platform.environment['LIVE_FEEDS'] == '1';
@@ -29,6 +33,19 @@ void main() {
expect(failures, isEmpty);
}, skip: live ? false : 'set LIVE_FEEDS=1 to run', timeout: const Timeout(Duration(minutes: 3)));
test('a full refresh with all on-by-default Sources stays fast', () async {
SharedPreferences.setMockInitialValues({});
final state = AppState(await Storage.open(), feedService: FeedService());
final clock = Stopwatch()..start();
await state.init();
clock.stop();
// ignore: avoid_print
print('${state.enabledSources.length} Sources, ${state.articles.length} articles '
'(${state.articlesOf(SourceKind.learning).length} Learning) in '
'${clock.elapsedMilliseconds} ms; errors: ${state.feedErrors}');
expect(clock.elapsed, lessThan(const Duration(seconds: 15)));
}, skip: live ? false : 'set LIVE_FEEDS=1 to run', timeout: const Timeout(Duration(minutes: 1)));
test('reader view extracts the article body', () async {
final service = FeedService();
final extractor = ArticleExtractor(consent: ConsentService());
+75
View File
@@ -0,0 +1,75 @@
import 'dart:io';
import 'package:daily_tech_news/models/article.dart';
import 'package:daily_tech_news/models/news_source.dart';
import 'package:daily_tech_news/services/feed_parser.dart';
import 'package:daily_tech_news/services/feed_service.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'support/fake_app.dart';
/// Serves saved Substack feeds (trimmed real items) instead of the network,
/// dated an hour ago so every time window keeps them.
class _SavedFeeds extends FeedService {
_SavedFeeds(this.fixtureByFeedUrl);
final Map<String, String> fixtureByFeedUrl;
@override
Future<List<Article>> fetchSource(NewsSource source, {String corsProxy = ''}) async {
final xml = File(fixtureByFeedUrl[source.feedUrl]!).readAsStringSync();
final anHourAgo = DateTime.now().toUtc().subtract(const Duration(hours: 1));
return [
for (final a in FeedParser.parse(xml, source))
Article(
title: a.title,
link: a.link,
source: a.source,
published: anHourAgo,
excerpt: a.excerpt,
contentHtml: a.contentHtml,
),
];
}
}
List<String> _titles(List<Article> articles) => articles.map((a) => a.title).toList();
void main() {
final interconnects = testSource('interconnects', SourceKind.learning);
final substackNews = testSource('substack-news', SourceKind.news);
setUp(() => SharedPreferences.setMockInitialValues({}));
_SavedFeeds savedFeeds() => _SavedFeeds({
interconnects.feedUrl: 'test/fixtures/interconnects_feed.xml',
substackNews.feedUrl: 'test/fixtures/ahead_of_ai_feed.xml',
});
test('paid newsletter posts are hidden in the Learn tab and Today and counted as hidden',
() async {
final state = await launchApp([interconnects, substackNews], feedService: savedFeeds());
expect(_titles(state.articlesOf(SourceKind.learning)),
['When will average people feel AI’s impact?']);
expect(state.hiddenCountOf(SourceKind.learning), 1);
expect(_titles(state.articlesOf(SourceKind.news)), [
'Controlling Reasoning Effort in LLMs',
'GPT-6 Astra, Looped Transformers, and Hidden Reasoning',
]);
expect(state.hiddenCountOf(SourceKind.news), 1);
});
test('with "hide paid articles" off, paid newsletter posts show as before', () async {
final state = await launchApp([interconnects, substackNews], feedService: savedFeeds());
await state.updateSettings(state.settings.copyWith(hidePaid: false));
expect(_titles(state.articlesOf(SourceKind.learning)).toSet(),
{'Lessons from the hacks', 'When will average people feel AI’s impact?'});
expect(state.hiddenCountOf(SourceKind.learning), 0);
expect(_titles(state.articlesOf(SourceKind.news)), contains('Building an AI Text Detector From Scratch'));
expect(state.hiddenCountOf(SourceKind.news), 0);
});
}
+18
View File
@@ -1,5 +1,8 @@
import 'dart:io';
import 'package:daily_tech_news/models/article.dart';
import 'package:daily_tech_news/models/news_source.dart';
import 'package:daily_tech_news/services/feed_parser.dart';
import 'package:daily_tech_news/services/paywall_service.dart';
import 'package:daily_tech_news/services/storage.dart';
import 'package:flutter_test/flutter_test.dart';
@@ -38,6 +41,21 @@ void main() {
expect(paywall.isSponsored(_article('Anzeigenmarkt bricht ein')), isFalse);
});
test('recognizes paid Substack posts by the cut-off content in the feed', () async {
final paywall = PaywallService(await _storage(), client: MockClient((_) async => http.Response('', 500)));
final posts = {
for (final a in FeedParser.parse(
File('test/fixtures/ahead_of_ai_feed.xml').readAsStringSync(), _source()))
a.title: a,
};
expect(paywall.isPaid(posts['Building an AI Text Detector From Scratch']!), isTrue);
expect(paywall.isPaid(posts['Controlling Reasoning Effort in LLMs']!), isFalse,
reason: 'a free post asking readers to become paid subscribers');
expect(paywall.isPaid(posts['GPT-6 Astra, Looped Transformers, and Hidden Reasoning']!), isFalse,
reason: 'a free post ending with the subscribe widget');
});
test('checks pages only for flagged sources and remembers results', () async {
final storage = await _storage();
final requested = <String>[];
+158
View File
@@ -0,0 +1,158 @@
import 'dart:convert';
import 'package:daily_tech_news/data/default_sources.dart';
import 'package:daily_tech_news/models/article.dart';
import 'package:daily_tech_news/models/briefing.dart';
import 'package:daily_tech_news/models/news_source.dart';
import 'package:daily_tech_news/services/summarizer.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'support/fake_app.dart';
/// Remembers which articles a Briefing was asked for.
class _RecordingSummarizer implements Summarizer {
List<Article>? briefingInput;
@override
String get name => 'test';
@override
Future<String> summarizeArticle(Article article, String text, SourceLanguage language) async =>
'';
@override
Future<DailyBriefing> createBriefing(List<Article> articles, SourceLanguage language) async {
briefingInput = articles;
return DailyBriefing(createdAt: DateTime.now(), intro: '', topics: const [], generatedBy: name);
}
}
List<String> _titles(List<Article> articles) => articles.map((a) => a.title).toList();
void main() {
final verge = testSource('verge', SourceKind.news);
final bytebytego = testSource('bytebytego', SourceKind.learning);
setUp(() => SharedPreferences.setMockInitialValues({}));
test('News uses the 24 h setting and Learning reaches back 14 days', () async {
final state = await launchApp([verge, bytebytego], feeds: {
verge.feedUrl: [('News from this morning', const Duration(hours: 2)),
('News from yesterday', const Duration(hours: 30))],
bytebytego.feedUrl: [('Explainer from 2 days ago', const Duration(days: 2)),
('Explainer from 13 days ago', const Duration(days: 13)),
('Explainer from 15 days ago', const Duration(days: 15))],
});
expect(_titles(state.articlesOf(SourceKind.news)), ['News from this morning']);
expect(_titles(state.articlesOf(SourceKind.learning)),
['Explainer from 2 days ago', 'Explainer from 13 days ago']);
});
test('the Briefing is generated from News articles only', () async {
final summarizer = _RecordingSummarizer();
final state = await launchApp([verge, bytebytego], feeds: {
verge.feedUrl: [('News from this morning', const Duration(hours: 2))],
bytebytego.feedUrl: [('Explainer from today', const Duration(hours: 1))],
}, summarizer: summarizer);
await state.generateBriefing();
expect(_titles(summarizer.briefingInput!), ['News from this morning']);
});
test('News follows the window setting', () async {
SharedPreferences.setMockInitialValues({'max_age_hours': 48});
final state = await launchApp([verge], feeds: {
verge.feedUrl: [('News from yesterday', const Duration(hours: 30)),
('News from 3 days ago', const Duration(hours: 72))],
});
expect(_titles(state.articlesOf(SourceKind.news)), ['News from yesterday']);
});
test('Fun reaches back 3 days', () async {
final xkcd = testSource('xkcd', SourceKind.fun);
final state = await launchApp([xkcd], feeds: {
xkcd.feedUrl: [('Comic from 2 days ago', const Duration(days: 2)),
('Comic from 4 days ago', const Duration(days: 4))],
});
expect(_titles(state.articlesOf(SourceKind.fun)), ['Comic from 2 days ago']);
});
test('custom Sources saved before kinds existed load as News', () async {
SharedPreferences.setMockInitialValues({
'custom_sources': jsonEncode([
{'id': 'custom-1', 'name': 'My blog', 'feedUrl': 'https://blog.example.com/feed',
'homepage': 'https://blog.example.com', 'language': 'de', 'description': ''},
]),
});
final state = await launchApp([verge]);
expect(state.customSources.single.kind, SourceKind.news);
});
test('a custom Learning source is still a Learning source after a restart', () async {
final before = await launchApp([verge]);
await before.addCustomSource(
name: 'Engineering blog',
feedUrl: 'https://eng.example.com/feed',
language: SourceLanguage.en,
kind: SourceKind.learning,
);
final after = await launchApp([verge]);
expect(after.customSources.single.name, 'Engineering blog');
expect(after.customSources.single.kind, SourceKind.learning);
});
test('paid and sponsored Learning articles are hidden like News ones', () async {
final state = await launchApp([bytebytego], feeds: {
bytebytego.feedUrl: [('How DNS works', const Duration(days: 1)),
('Sponsored: Try our database', const Duration(days: 1))],
});
expect(_titles(state.articlesOf(SourceKind.learning)), ['How DNS works']);
expect(state.hiddenCountOf(SourceKind.learning), 1);
});
test('every curated Source from before kinds is a News source', () {
final older = defaultSources.where((s) => legacySeenSourceIds.contains(s.id));
expect(older, isNotEmpty);
expect(older.where((s) => s.kind != SourceKind.news), isEmpty);
});
test('ByteByteGo is a Learning source that is on by default', () {
final bytebytego = defaultSources.singleWhere((s) => s.name == 'ByteByteGo');
expect(bytebytego.kind, SourceKind.learning);
expect(bytebytego.feedUrl, 'https://blog.bytebytego.com/feed');
expect(bytebytego.language, SourceLanguage.en);
expect(bytebytego.enabledByDefault, isTrue);
});
test('a fresh install shows posts of the last 14 days from the curated Learning sources',
() async {
final learning = defaultSources.where((s) => s.kind == SourceKind.learning);
final state = await launchApp(defaultSources, feeds: {
for (final s in learning)
s.feedUrl: [('${s.name}: recent', const Duration(days: 13)),
('${s.name}: old', const Duration(days: 15))],
});
expect(_titles(state.articlesOf(SourceKind.learning)).toSet(), {
'ByteByteGo: recent', 'System Design One: recent',
'The Pragmatic Engineer (blog): recent', 'Martin Fowler: recent', 'Julia Evans: recent',
'Dan Luu: recent', 'The Cloudflare Blog: recent', 'Netflix Technology Blog: recent',
'Engineering at Meta: recent', 'GitHub Blog: Engineering: recent', 'Latent Space: recent',
'Import AI: recent', 'Interconnects: recent', 'Ahead of AI: recent',
'Lilian Weng: recent',
// InfoQ is off by default.
});
});
}
+13 -3
View File
@@ -10,7 +10,7 @@ import 'package:shared_preferences/shared_preferences.dart';
class _NoFeeds extends FeedService {
@override
Future<FeedResult> fetchAll(List<NewsSource> sources,
{required Duration maxAge, String corsProxy = ''}) async =>
{required Duration Function(NewsSource source) maxAgeOf, String corsProxy = ''}) async =>
const FeedResult([], {});
}
@@ -23,6 +23,13 @@ NewsSource _source(String id, {bool onByDefault = false}) => NewsSource(
enabledByDefault: onByDefault,
);
/// The curated Learning sources that are on by default (all but InfoQ).
const _learningOnByDefault = {
'bytebytego', 'systemdesignone', 'pragmaticengineer-blog', 'martinfowler', 'juliaevans',
'danluu', 'cloudflare-blog', 'netflix-techblog', 'meta-engineering', 'github-engineering',
'latent-space', 'import-ai', 'interconnects', 'ahead-of-ai', 'lilianweng',
};
/// Starts the app the way a launch does, against whatever is in preferences.
Future<AppState> _launch(List<NewsSource> curatedSources) async {
final state =
@@ -87,7 +94,8 @@ void main() {
expect(state.enabledSourceIds, {
'heise', 'verge', 'mit-tr', 'new-source',
// Added to the catalog later, on by default.
'techmeme', 'lobsters', 'the-decoder', 'simonwillison',
'techmeme', 'lobsters', 'the-decoder', 'simonwillison', ..._learningOnByDefault,
'xkcd', 'thedailywtf',
});
});
@@ -102,8 +110,10 @@ void main() {
expect(state.enabledSourceIds, {
'heise', 'golem', 't3n', 'verge',
'techmeme', 'lobsters', 'the-decoder', 'simonwillison',
'techmeme', 'lobsters', 'the-decoder', 'simonwillison', ..._learningOnByDefault,
'xkcd', 'thedailywtf',
});
expect(state.enabledSources.map((s) => s.id), isNot(contains('wired')));
expect(state.enabledSourceIds, isNot(contains('infoq')));
});
}
+184
View File
@@ -0,0 +1,184 @@
import 'package:daily_tech_news/models/news_source.dart';
import 'package:daily_tech_news/state/app_state.dart';
import 'package:daily_tech_news/ui/home_shell.dart';
import 'package:daily_tech_news/ui/learn_screen.dart';
import 'package:daily_tech_news/ui/sources_screen.dart';
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'support/fake_app.dart';
NewsSource _source(String id, SourceKind kind, SourceLanguage language) => NewsSource(
id: id,
name: id,
feedUrl: 'https://example.com/$id.xml',
homepage: 'https://example.com',
language: language,
kind: kind,
enabledByDefault: true,
);
void main() {
final heise = _source('heise', SourceKind.news, SourceLanguage.de);
final verge = _source('verge', SourceKind.news, SourceLanguage.en);
final bytebytego = _source('bytebytego', SourceKind.learning, SourceLanguage.en);
final xkcd = _source('xkcd', SourceKind.fun, SourceLanguage.en);
const german = '🇩🇪 Deutsch';
const english = '🇬🇧 English';
setUp(() => SharedPreferences.setMockInitialValues({}));
Future<AppState> launch(WidgetTester tester, List<NewsSource> sources) async =>
(await tester.runAsync(() => launchApp(sources)))!;
Future<void> showSources(WidgetTester tester, AppState state) async {
tester.view.physicalSize = const Size(800, 2400);
tester.view.devicePixelRatio = 1;
addTearDown(tester.view.reset);
await tester.pumpWidget(
MaterialApp(home: SourcesScreen(state: state, onChanged: () {})));
}
/// Asserts the [finders] each match once on screen, from top to bottom.
void expectInOrder(WidgetTester tester, List<Finder> finders) {
final ys = [for (final f in finders) tester.getTopLeft(f).dy];
expect(ys, [...ys]..sort());
expect(ys.toSet().length, ys.length);
}
testWidgets('the Sources tab groups News, Learning and Fun, each split by language',
(tester) async {
final state = await launch(tester, [xkcd, bytebytego, verge, heise]);
await showSources(tester, state);
expectInOrder(tester, [
find.text('News'),
find.text(german).at(0),
find.text('heise'),
find.text(english).at(0),
find.text('verge'),
find.text('Learning'),
find.text(english).at(1),
find.text('bytebytego'),
find.text('Fun'),
find.text(english).at(2),
find.text('xkcd'),
]);
});
testWidgets('a kind without Sources gets no section', (tester) async {
final state = await launch(tester, [verge]);
await showSources(tester, state);
expect(find.text('News'), findsOneWidget);
expect(find.text('Learning'), findsNothing);
expect(find.text('Fun'), findsNothing);
expect(find.text(german), findsNothing);
});
Future<void> addFeed(WidgetTester tester, String name, {String? kind}) async {
await tester.tap(find.text('Add feed'));
await tester.pumpAndSettle();
final dialog = find.byType(AlertDialog);
await tester.enterText(
find.descendant(of: dialog, matching: find.widgetWithText(TextField, 'Name')), name);
await tester.enterText(
find.descendant(of: dialog, matching: find.widgetWithText(TextField, 'Feed URL')),
'https://example.com/$name.xml');
if (kind != null) {
await tester.tap(find.descendant(of: dialog, matching: find.text(kind)));
await tester.pump();
}
await tester.tap(find.descendant(of: dialog, matching: find.text('Add')));
await tester.pumpAndSettle();
}
testWidgets('a feed added without picking a kind is a News source', (tester) async {
final state = await launch(tester, [verge, bytebytego]);
await showSources(tester, state);
await addFeed(tester, 'myfeed');
expectInOrder(tester, [find.text('News'), find.text('myfeed'), find.text('Learning')]);
});
testWidgets('a feed added as Learning shows up in the Learning section', (tester) async {
final state = await launch(tester, [verge, bytebytego]);
await showSources(tester, state);
await addFeed(tester, 'myblog', kind: 'Learning');
expectInOrder(tester, [find.text('Learning'), find.text('myblog')]);
});
testWidgets('a feed added as Fun gets a Fun section', (tester) async {
final state = await launch(tester, [verge]);
await showSources(tester, state);
await addFeed(tester, 'mycomic', kind: 'Fun');
expectInOrder(tester, [find.text('News'), find.text('Fun'), find.text('mycomic')]);
});
testWidgets('a custom Learning source shows up in the Learn tab, not in Today',
(tester) async {
final feeds = FakeFeeds({
verge.feedUrl: [('News from this morning', const Duration(hours: 2))],
'https://example.com/myblog.xml': [('Explainer from my blog', const Duration(days: 2))],
});
final state = (await tester.runAsync(() => launchApp([verge], feedService: feeds)))!;
tester.view.physicalSize = const Size(800, 2400);
tester.view.devicePixelRatio = 1;
addTearDown(tester.view.reset);
await tester.pumpWidget(MaterialApp(home: HomeShell(state: state)));
Finder tab(String label) =>
find.descendant(of: find.byType(NavigationBar), matching: find.text(label));
await tester.tap(tab('Sources'));
await tester.pump();
await addFeed(tester, 'myblog', kind: 'Learning');
await tester.tap(tab('Learn'));
await tester.pumpAndSettle();
expect(find.text('Explainer from my blog'), findsOneWidget);
await tester.tap(tab('Today'));
await tester.pumpAndSettle();
expect(find.text('News from this morning'), findsOneWidget);
expect(find.text('Explainer from my blog'), findsNothing);
});
testWidgets('a custom Fun source shows up in the Fun section of the Learn tab, not in Today',
(tester) async {
final feeds = FakeFeeds({
verge.feedUrl: [('News from this morning', const Duration(hours: 2))],
'https://example.com/mycomic.xml': [('Comic from my feed', const Duration(days: 1))],
});
final state = (await tester.runAsync(() => launchApp([verge], feedService: feeds)))!;
tester.view.physicalSize = const Size(800, 2400);
tester.view.devicePixelRatio = 1;
addTearDown(tester.view.reset);
await tester.pumpWidget(MaterialApp(home: HomeShell(state: state)));
Finder tab(String label) =>
find.descendant(of: find.byType(NavigationBar), matching: find.text(label));
Finder onLearn(String text) =>
find.descendant(of: find.byType(LearnScreen), matching: find.text(text));
await tester.tap(tab('Sources'));
await tester.pump();
await addFeed(tester, 'mycomic', kind: 'Fun');
await tester.tap(tab('Learn'));
await tester.pumpAndSettle();
expectInOrder(tester, [onLearn('Learning'), onLearn('Fun'), onLearn('Comic from my feed')]);
await tester.tap(tab('Today'));
await tester.pumpAndSettle();
expect(find.text('News from this morning'), findsOneWidget);
expect(find.text('Comic from my feed'), findsNothing);
});
}
+71
View File
@@ -0,0 +1,71 @@
import 'package:daily_tech_news/models/article.dart';
import 'package:daily_tech_news/models/news_source.dart';
import 'package:daily_tech_news/services/article_extractor.dart';
import 'package:daily_tech_news/services/feed_service.dart';
import 'package:daily_tech_news/services/storage.dart';
import 'package:daily_tech_news/services/summarizer.dart';
import 'package:daily_tech_news/state/app_state.dart';
/// Serves canned articles per Source instead of the network. The real
/// [FeedService.fetchAll] still applies the time windows.
class FakeFeeds extends FeedService {
FakeFeeds([this.byFeedUrl = const {}]);
/// Feed URL -> (title, age) of each article in the feed.
final Map<String, List<(String, Duration)>> byFeedUrl;
/// How often the app refreshed its feeds.
int refreshes = 0;
@override
Future<FeedResult> fetchAll(List<NewsSource> sources,
{required Duration Function(NewsSource source) maxAgeOf, String corsProxy = ''}) {
refreshes++;
return super.fetchAll(sources, maxAgeOf: maxAgeOf, corsProxy: corsProxy);
}
@override
Future<List<Article>> fetchSource(NewsSource source, {String corsProxy = ''}) async {
final now = DateTime.now().toUtc();
// A feed nobody set up still has an article, so it can be added.
if (!byFeedUrl.containsKey(source.feedUrl)) {
return [Article(title: 'Any', link: '${source.feedUrl}#any', source: source)];
}
return [
for (final (title, age) in byFeedUrl[source.feedUrl]!)
Article(
title: title,
link: '${source.feedUrl}#$title',
source: source,
published: now.subtract(age),
),
];
}
}
NewsSource testSource(String id, SourceKind kind) => NewsSource(
id: id,
name: id,
feedUrl: 'https://example.com/$id.xml',
homepage: 'https://example.com',
language: SourceLanguage.en,
kind: kind,
enabledByDefault: true,
);
/// Starts the app the way a launch does, against whatever is in preferences.
Future<AppState> launchApp(
List<NewsSource> curatedSources, {
Map<String, List<(String, Duration)>> feeds = const {},
FeedService? feedService,
Summarizer? summarizer,
ArticleExtractor? extractor,
}) async {
final state = AppState(await Storage.open(),
feedService: feedService ?? FakeFeeds(feeds),
curatedSources: curatedSources,
summarizer: summarizer,
extractor: extractor);
await state.init();
return state;
}