Download

Write a source

The source

A source answers Katari's questions about content.

The factory #

The class named by tachiyomi.extension.class creates the extension's sources. A site with one catalogue per language returns one source for each.

Kotlin
class ExampleComicsFactory : EntrySourceFactory {
    override fun createSources(): List<UnifiedSource> = listOf(ExampleComics())
}

Katari creates the factory when it loads the extension. An exception thrown there, or in a source's constructor, fails the whole extension, so leave network requests for later.

The source class #

Sources extend EntryHttpSource, or EntryImageHttpSource for comics. A source names itself, its site and its language, then implements the methods in the sections below and getMedia, which Pages and books and Video cover.

Kotlin
class ExampleComics : EntryImageHttpSource(), SourceMetadata {
    override val id = 4316702938517726312L
    override val name = "Example Comics"
    override val lang = "en"
    override val baseUrl = "https://comics.example"
    override val supportsLatest = true
    override val supportedEntryTypes = setOf(EntryType.MANGA)

    override val client: OkHttpClient by lazy {
        network.client.newBuilder().rateLimit(2).build()
    }

    private suspend fun fetch(url: String): Document =
        client.newCall(GET(url, headers)).awaitSuccess().asJsoup()
  • lang is a language code such as en, pt-BR or zh-Hans; all for a source in several languages, or other for one in none. Users see sources in English, in the system's language and under all at first, matched by the bare language code, so a pt-BR source starts hidden even on a system in Brazilian Portuguese, and turn other languages on themselves.
  • supportedEntryTypes comes from the optional SourceMetadata and tells Katari which kinds of title to expect before any arrive, for the badges and the Content types filter. Without it, or with an empty set, the source counts as unspecified. Each title's own type still decides how it opens.
  • id is the source's identity: library titles belong to it, and your store's index lists it. Pick a random positive number once, such as with this command, and never change it:
    Shell
    python3 -c "import secrets; print(secrets.randbits(63))"

    Left out, it is made from the lowercased name, the language and versionId, and renaming the source would leave library titles pointing at a source that no longer exists.

    Two sources with the same ID are the same source to Katari, even in different extensions, so a port of a Mihon source keeps its users' library by keeping its ID. When two installed extensions provide one ID, Katari keeps using the first, and the user can switch with Use this one.

EntryHttpSource fills in the rest, and each of these can be overridden:

val supportsLatest: Boolean = false
The Latest list shows only when this is true.
open val versionId: Int = 1
Only used to make an id that was left out.
protected open fun headersBuilder(): Headers.Builder
Returns a builder with only User-Agent, from the user's setting under Settings › Advanced. Override it to add headers the site wants with every request, such as Referer.
open val headers: Headers
Built from headersBuilder() once, the first time it is read. Pass it to every request. Katari sends it with covers and chapter stills too.
open val client: OkHttpClient
network.client; see Requests. Katari loads covers and chapter stills with it too.
fun getHomeUrl(): String
Returns baseUrl, which WebView opens from the source's list.
fun getContentUrl(entry: SEntry): String
Returns baseUrl + entry.url, the title's page in WebView.
fun getWebViewHeaders(): Map<String, String>
Returns the first value of each of headers, sent when WebView opens a page.

EntryImageHttpSource adds what the comic reader and downloader need:

fun getChapterUrl(chapter: SEntryChapter): String
Returns baseUrl + chapter.url, the chapter's page in WebView.
suspend fun getImageUrl(page: EntryImagePage): String
Returns the page's imageUrl, or its url when that is unset.
fun imageRequest(page: EntryImagePage, imageUrl: String): Request
Returns GET(imageUrl, headers).

Both give network, and setUrlWithoutDomain for titles, which keeps a full address's path, query and fragment. EntryImageHttpSource also has it for chapters; other sources call getUrlWithoutDomain, which returns the same.

Popular, latest and search #

Kotlin
override suspend fun getPopularContent(page: Int) = listing("$baseUrl/popular?page=$page")

override suspend fun getLatestUpdates(page: Int) = listing("$baseUrl/latest?page=$page")

override suspend fun getSearchContent(
    page: Int,
    query: String,
    filters: EntryFilterList,
): EntryPageResult<SEntry> {
    val url = "$baseUrl/search".toHttpUrl().newBuilder()
        .addQueryParameter("q", query)
        .addQueryParameter("page", page.toString())
        .apply { filters.applyTo(this) }
        .build()
    return listing(url.toString())
}

private suspend fun listing(url: String): EntryPageResult<SEntry> {
    val document = fetch(url)
    val entries = document.select(".comic-card").map { card ->
        SEntry.create().apply {
            setUrlWithoutDomain(card.selectFirst("a")!!.absUrl("href"))
            title = card.selectFirst(".title")!!.text()
            thumbnailUrl = card.selectFirst("img")?.absUrl("src")
            type = EntryType.MANGA
        }
    }
    return EntryPageResult(entries, hasNextPage = document.selectFirst(".pagination .next") != null)
}
  • Pages are numbered from 1. Return the page's titles as EntryPageResult(items, hasNextPage), and Katari asks for the next page as the user scrolls while hasNextPage is true. A page with no titles ends the list.
  • Feeds in Browse › Feeds also load pages by themselves: refreshing a chronological feed, such as Latest, keeps loading until it reaches titles it already has, up to 10 pages. List Latest newest first and keep its pages stable.
  • Set type on every title: MANGA, ANIME or BOOK. It is MANGA unless set, so a video or book left without one opens in the comic reader.
  • url is the title's identity: Katari stores it with each library title and passes it back in every later call. Keep it without the domain, so a site that moves only needs a new baseUrl, and never change its shape once published.
  • A listing needs only the title, address and cover. The rest comes from the details.
  • applyTo is the source's own function that turns filters into the site's parameters, as Filters shows.
  • When the site's search box ignores the filters, override textSearchUsesFilters to return false. While a query is typed, Katari then marks the filters as not used with a text search, so users don't expect them to apply.

Refreshing a title #

Katari calls refresh for a title's details, its chapters, or both. Load the page once and fill in what fetchDetails and fetchChapters ask for:

Kotlin
override suspend fun refresh(
    entry: SEntry,
    existingChapters: List<SEntryChapter>,
    fetchDetails: Boolean,
    fetchChapters: Boolean,
): EntryRefresh {
    val document = fetch(baseUrl + entry.url)
    return EntryRefresh(
        entry = if (fetchDetails) details(entry, document) else entry,
        chapters = if (fetchChapters) chapters(document) else existingChapters,
    )
}
  • Return what wasn't asked for unchanged: entry as it came, or existingChapters. Never return a placeholder; Katari may use it.
  • existingChapters holds the chapters Katari has stored. A site that lists only new chapters can add them to it and return the whole list. Don't modify the chapters passed in, since two refreshes of one title can run at once.

Details #

Kotlin
private fun details(entry: SEntry, document: Document): SEntry =
    entry.copy().apply {
        title = document.selectFirst("h1")!!.text()
        author = document.selectFirst(".author")?.text()
        description = document.selectFirst(".synopsis")?.text()
        genre = document.select(".tags a").map { it.text() }
        status = when (document.selectFirst(".status")?.text()) {
            "Ongoing" -> SEntry.ONGOING
            "Completed" -> SEntry.COMPLETED
            else -> SEntry.UNKNOWN
        }
    }
Field Holds When the details leave it empty
url The title's address, as in the listing. Katari ignores the field and keeps its own.
type MANGA, ANIME or BOOK. Katari ignores the field and keeps its own.
title The name. Keeps the old one. A new name replaces a library title's only while Update library entry titles to match source is on, and never for a season of a merged entry.
author, artist, description Text, or null. Keeps the old one.
genre A list of tags, or null. Keeps the old one.
thumbnailUrl The cover's full address, or null. Keeps the old one.
backgroundUrl The full address of wide art shown behind the title's header, or null. Keeps the old one.
status One of the constants below. Becomes unknown.
updateStrategy EntryUpdateStrategy.ALWAYS_UPDATE, or ONLY_FETCH_ONCE for a title whose content never changes, such as a finished book: library updates skip it once it has chapters. Keeps the old one, which starts as ALWAYS_UPDATE.
merged true for a merged entry. Becomes false.
memo A JsonObject the user never sees, for the source's own use. Katari stores it and passes it back. Becomes empty.

status is SEntry.UNKNOWN, ONGOING, COMPLETED, LICENSED, PUBLISHING_FINISHED, CANCELLED, ON_HIATUS or UPCOMING for a title announced but not out yet. Katari sets initialized itself.

Start from a copy of the title Katari passes in, as the snippet does. The details then keep its memo and whatever the page doesn't show again.

Chapters #

Chapters are what a title is made of, whatever its kind: a comic's chapters, a show's episodes, a novel's chapters or a single book.

Kotlin
private fun chapters(document: Document): List<SEntryChapter> =
    document.select(".chapters li a").map { link ->
        SEntryChapter.create().apply {
            setUrlWithoutDomain(link.absUrl("href"))
            name = link.text()
            dateUpload = link.selectFirst("time")?.attr("datetime")
                ?.let { Instant.parse(it).toEpochMilli() } ?: 0L
        }
    }
var url: String
The chapter's identity within the title, without the domain. Katari passes it back to getMedia. A repeated one is dropped, keeping the first.
var name: String
Shown in the chapter list. A blank name shows the url instead.
var dateUpload: Long
Milliseconds since 1970, or 0 when unknown.
var dateText: String?
The release time as the site words it, such as 3 days ago or Spring 2024, for a chapter whose date can't be read. It is shown in place of the date, so leave it null whenever dateUpload is set.
var chapterNumber: Double
-1.0 when unknown. Add ChapterNumberRecognitionSource to the source to let Katari read unknown numbers from the chapters' names.
var scanlator: String?
The team, translator or provider that released it, or null. Users can filter chapters by it.
var summary: String?
A short synopsis, shown under the chapter's name, or null.
var previewUrl: String?
The full address of a still or preview image, shown at the start of the chapter's row, or null. It is loaded with the source's headers.
var kind: ChapterKind
REGULAR, the default, or FILLER for a chapter or episode outside the main story. Users can hide filler.
var group: String?
The key of the volume or other part of the story the chapter belongs to, or null. See Chapter groups.
var memo: JsonObject
For the source's own use, stored and passed back with the chapter.

List the chapters newest first, as most sites do. Katari keeps this order when the user sorts by source.

Chapter groups #

A site that sorts a title's chapters into volumes, arcs or books can say so. Give each chapter its group's key, and list the groups in chapterGroups:

Kotlin
// In refresh(), for a page that lists its volumes oldest first
val volumes = document.select(".volume")
val listed = volumes.flatMap { volume ->
    volume.select("li a").map { link ->
        SEntryChapter.create().apply {
            url = getUrlWithoutDomain(link.absUrl("href"))
            name = link.text()
            group = volume.attr("data-id")
        }
    }
}
val groups = volumes.map { volume ->
    SEntryChapterGroup.create(key = volume.attr("data-id"), title = volume.selectFirst("h2")!!.text())
}
return EntryRefresh(entry, listed.reversed(), chapterGroups = groups)
  • Katari shows each group as a section headed by its title, sorts and reads chapters group by group in the order of chapterGroups, and counts duplicate and missing chapter numbers within each group, so numbering may start over in every volume.
  • The key ties chapters to their group and keeps the user's open and closed sections; keep it when the title changes. A chapter without a group, or with a key the refresh doesn't list, goes under Other chapters.
  • A group with no chapters yet, such as an announced volume, shows as empty.
  • Katari reads chapterGroups only when fetchChapters asked for chapters.

A group is a part of the story. A translation team, branch or dub is the chapter's scanlator instead, and a part with a page and details of its own is a title of its own, offered through a merged entry.

Long chapter lists #

When a site lists a title's chapters over many pages, reading them all in refresh keeps the user waiting. Add ChapterListPartsSource and return the newest page with what follows it; Katari shows those chapters at once and reads the older pages in the background, one at a time, through getOlderChapters:

Kotlin
class ExampleNovels : EntryHttpSource(), ChapterListPartsSource {
    override suspend fun refresh(
        entry: SEntry,
        existingChapters: List<SEntryChapter>,
        fetchDetails: Boolean,
        fetchChapters: Boolean,
    ): EntryRefresh {
        val details = if (fetchDetails) details(entry, fetch(baseUrl + entry.url)) else entry
        if (!fetchChapters) return EntryRefresh(details, existingChapters)
        val newest = chapterPage(entry, 1)
        // Stored chapters that reach this page's oldest one mean nothing older changed
        val known = existingChapters.any { it.url == newest.chapters.lastOrNull()?.url }
        return EntryRefresh(details, newest.chapters, rest = if (known) ChapterListRest.Unchanged else newest.rest)
    }

    override suspend fun getOlderChapters(entry: SEntry, token: String): ChapterListPart =
        chapterPage(entry, token.toInt())

    private suspend fun chapterPage(entry: SEntry, page: Int): ChapterListPart {
        val document = fetch("$baseUrl${entry.url}/chapters?page=$page")
        val pages = document.select(".pagination a.page").size
        return ChapterListPart(
            chapters = chapters(document),
            rest = if (page < pages) {
                ChapterListRest.More(token = "${page + 1}", loaded = page, total = pages)
            } else {
                ChapterListRest.Complete
            },
        )
    }
}
data object Complete : ChapterListRest
Nothing older follows. Only when a list ends with it does Katari remove stored chapters the list no longer has.
class More(token: String, loaded: Int? = null, total: Int? = null) : ChapterListRest
Older chapters follow, and getOlderChapters receives token. Katari stores the token so the list goes on after a restart, so keep it valid that long; it can be up to 4,096 characters. loaded and total count the site's pages, shown as Loading older chapters · 23 of 46 pages.
data object Unchanged : ChapterListRest
Nothing older changed: Katari keeps the stored chapters this part doesn't list, and an unfinished list goes on in the background. Answer it from refresh when the stored chapters show it, so a library update reads only the newest page.
  • List each part newest first, like a whole list. A chapter an earlier part already listed is ignored, so pages that shift while new chapters come out are safe.
  • A part declares the groups its chapters belong to in its own chapterGroups, even a group an earlier part declared.
  • Throw from getOlderChapters when a page fails. The chapter list shows why, with Retry, and Katari tries the same token again later.
  • Migrating a title, and downloading all, unread or the next chapters, wait until the whole list is read. Library updates finish lists that stopped.
  • More from a source without ChapterListPartsSource counts as Unchanged.

Merged entries #

A site that lists a show's seasons as separate titles can offer the show as one merged entry, the way users merge titles themselves. Set merged = true in the show's details, and add MergedEntrySource to list the seasons:

Kotlin
class ExampleShows : EntryHttpSource(), MergedEntrySource {
    // In details(): a show page that lists seasons is a merged entry
    //     merged = document.select(".seasons a").isNotEmpty()

    override suspend fun getMergedEntries(entry: SEntry): List<SEntry> =
        fetch(baseUrl + entry.url).select(".seasons a").map { link ->
            SEntry.create().apply {
                setUrlWithoutDomain(link.absUrl("href"))
                title = link.text()
                thumbnailUrl = link.selectFirst("img")?.absUrl("src")
                type = EntryType.ANIME
            }
        }
  • A merged entry has no chapters of its own. Katari asks getMergedEntries for its seasons instead, and they are ordinary titles refreshed like any other.
  • Katari decides whether a title is merged from its first details and doesn't change its mind, so base merged on something the page always shows.
  • List the seasons oldest first. Each must be a title of this source with the same type, and not merged itself. Katari skips any other.
  • Each season keeps the name you list it under, such as Season 1, even if its own page names the show.
  • An empty list counts as an error, so a page that fails to load can't empty the merge.

Filters #

The filters that getSearchContent receives are the controls of the search sheet, which the source declares in getFilterList. A source that doesn't override it has none. Filters shows how to declare and read them.

Requests #

network.client is the OkHttp client Katari uses itself. Build on it with newBuilder() rather than making a new one, and leave its socket factory and DNS alone. It has:

  • Katari's cookies, shared with WebView.
  • A 5 MiB cache, 30 second connect and read timeouts, and a 2 minute limit on a whole call.
  • The user's DNS over HTTPS choice from Settings › Advanced.
  • Katari's user agent on any request that has none.
  • Cloudflare's check: Katari passes it in a hidden WebView and repeats the request. Don't add a workaround of your own. When the check needs a person, the user sees Failed to bypass Cloudflare and solves it in WebView, as Troubleshooting describes.
  • AndroidAndroid's local network permission, which Katari asks for when a source connects to an address on the user's network. Without it, the connection fails with LocalNetworkAccessDeniedException, an IOException the user sees.

Requests are built and sent with these, from eu.kanade.tachiyomi.network:

fun GET(url: String, headers: Headers = DEFAULT_HEADERS, cache: CacheControl = DEFAULT_CACHE_CONTROL): Request
Another overload takes the url as an HttpUrl. headers is empty unless given, and cache accepts a cached response up to 10 minutes old unless you pass another CacheControl, such as CacheControl.FORCE_NETWORK.
fun POST(url: String, headers: Headers = DEFAULT_HEADERS, body: RequestBody = DEFAULT_BODY, cache: CacheControl = DEFAULT_CACHE_CONTROL): Request
The same, with a body that is an empty form unless given. PUT and DELETE take the same parameters.
suspend fun Call.await(): Response
Sends the request and returns the response, whatever its status.
suspend fun Call.awaitSuccess(): Response
The same, but closes a response whose status isn't 2xx and throws HttpException, whose message, such as HTTP error 404, the user sees.
suspend fun OkHttpClient.get(url: String, headers: Headers = DEFAULT_HEADERS, cache: CacheControl = DEFAULT_CACHE_CONTROL): Response
Short for newCall(GET(url, headers, cache)).awaitSuccess(). It also takes an HttpUrl, and post does the same for POST.
fun OkHttpClient.Builder.rateLimit(permits: Int, period: Duration = 1.seconds)
At most permits requests per period, a kotlin.time.Duration. rateLimit(2) allows two requests a second.
fun OkHttpClient.Builder.rateLimitHost(url: String, permits: Int, period: Duration = 1.seconds)
The same for requests to one host. A url that isn't a valid address limits every host.

Library updates and downloads ask for many titles at once, so set a rate limit even if browsing feels fine without one. Network failures reach the user as the IOException OkHttp throws.

For a site with an API, decode its JSON with the instance Katari provides:

Kotlin
private val json: Json by injectLazy()

private suspend fun search(url: String): SearchResponse {
    val body = client.newCall(GET(url, headers)).awaitSuccess().body.string()
    return json.decodeFromString(body)
}

Services #

injectLazy() and Injekt.get() return Katari's own instances of these types, and no others:

class NetworkHelper
Sources already have it as network.

Members

val client: OkHttpClient
Described in Requests.
fun defaultUserAgentProvider(): String
The user agent from Katari's settings.
sealed class Json
kotlinx.serialization's JSON with ignoreUnknownKeys = true, so fields the site adds later don't break decoding, and explicitNulls = false, so a missing nullable field reads as null.
sealed class ProtoBuf
kotlinx.serialization's default ProtoBuf.
abstract class JavaScriptEngine

Members

suspend fun <T> evaluate(script: String): T
Runs a script, such as a site's obfuscated player code, and returns its value as a String, an Int or Double, a Boolean, null, or an Array<Any?> of these for an array. Promise callbacks the script queues run before it returns. Each call starts a new QuickJS context, so nothing carries over from one to the next.
open class Application
Katari's application.
abstract class Context
Katari's application, as above.
interface EntryPreferenceProvider
Where getSourcePreferences() reads from. Use that function instead, as Settings shows.

Asking for any other type throws InjektionException. Katari's Injekt is read-only: registering a type of your own, or keyed and logger injection, throws UnsupportedOperationException. Create your own objects in the source instead.