Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .dev.vars.example
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,17 @@ SEED_MAX_METADATA_SIZE=10485760
# 兼容旧变量名 ALLOW_SEED(新变量优先)。
SEED_SOURCE_ALLOWED_HOSTS=

# ── 任务系统 ────────────────────────────────────────────────────────────────
# URL 离线下载的主机白名单,逗号/空格/分号分隔,支持 *.example.com。
# **默认留空 = 拒绝一切外网下载**(防止被当作免费代理刷流量)。
# 任务在执行时会重新校验该白名单,排队期间收紧配置也会生效。
ALLOW_OFFLINE_DOWNLOAD=

# 部署 origin(如 https://your-domain.com)。后台任务没有请求对象,存储驱动
# 返回的相对下载直链(/d/...)需要它才能解析。Cloudflare 部署通常不需要填
# (入队时会把当前请求的 origin 写进任务);EdgeOne / Node 建议显式配置。
PUBLIC_ORIGIN=

# 说明:seed_single_direct_preview 不是环境变量,而是数据库中的站点设置项
# (默认 false)。可在管理后台「设置 → 种子」中修改,单文件种子解析时生效。

Expand Down
8 changes: 8 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,11 @@ CF_API_KEY=
# 目录列表与下载直链每次都直接回源驱动,既不做跨请求缓存,也不合并并发相同
# 调用。相关环境变量(CACHE_* / SINGLEFLIGHT*)已全部删除,残留同名变量不会
# 再被读取。

# ── 任务系统 ────────────────────────────────────────────────────────────────
# URL 离线下载的主机白名单,逗号/空格/分号分隔,支持 *.example.com。
# 默认留空 = 拒绝一切外网下载。
ALLOW_OFFLINE_DOWNLOAD=

# 部署 origin(后台任务解析相对下载直链时需要)。
PUBLIC_ORIGIN=
40 changes: 40 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,10 +153,50 @@ pnpm run deploy:worker
| `DB_DRIVER` | `auto` | 可选 `auto` / `blob` / `cfkv` / `kv` / `d1` / `do` / `mysql` |
| `DB_CIPHER` | `none` | 敏感字段落盘算法,如 `aes-256-gcm` |

> **KV 配额提示**:`DB_FORMAT=key` 会维护全局索引键(`openlist_key_index`),
> 首次读取旧数据后 load/save 不再消耗 KV `list` 配额。但每次保存仍按记录数写入
> (默认配置约 100+ 条),Cloudflare KV 免费版只有 1000 写/天,长期使用建议
> `DB_DRIVER=d1` 或 `DB_FORMAT=map`。



完整环境变量,请参阅[配置变量](https://doc.oplist.org/ecosystem/official_worker/guide_env)

### 任务系统(异步操作)

大文件复制/移动不再阻塞请求:超过阈值(默认 32MB)的操作会转成持久化任务,
由 `TaskRunner` Durable Object 用 alarm 自驱执行,进度可在前端任务页查看。

- **调度**:Cloudflare 上由 `TASK_DO`(TaskRunner)驱动,入队即开始执行;每分钟的
cron 只是看门狗。没有 DO 绑定时(EdgeOne / ESA / Vercel / Node)自动降级为
「任务文档 + 请求内时间片」。
- **单次 tick 预算**:设置项 `task_tick_budget_ms`,默认 `0`(自动)。Cloudflare
自动值为 2000ms —— **免费计划的保守值**;付费计划建议改为 20000,
必要时可调到 60000。
- **并发**:`task_parallel_units` 默认 1,最大 4(Cloudflare 每次调用最多 6 条
出站连接,每个传输通常占 2 条)。
- **免费计划提醒**:Workers 免费计划 CPU 上限很低,大文件流式传输容易触发
CPU 超限;建议对大文件使用支持服务端直传(`putUrl`)的目标存储,或升级付费计划。
- **非 Cloudflare 平台**:EdgeOne 的 `schedules` 最小间隔为一天,无法作为分钟级
任务泵。请用外部 cron 每分钟请求一次:

```bash
curl -X POST https://your-domain/api/task/run \
-H "X-Cron-Secret: $JWT_SECRET"
```

该接口只执行已入队的任务,不接受任何路径/URL 参数。

任务相关设置(设置页「流量」分组):`task_enabled`、`task_sync_threshold_mb`、
`task_probe_max_entries`、`task_tick_budget_ms`、`task_parallel_units`、
`task_keep_done` 以及各类型的 `task_*_max_retry`。

**URL 离线下载**也走同一套任务队列(`POST /api/fs/add_offline_download`)。
出于安全考虑默认关闭:必须配置 `ALLOW_OFFLINE_DOWNLOAD`(主机白名单,
逗号分隔,支持 `*.example.com`),否则请求直接返回 403。目标存储没有流式/
分片上传能力时,单文件受内存上限约束(可用 `TASK_DOWNLOAD_MAX_BYTES` 调整,
默认 64MB)。


## 帮助支持

Expand Down
6 changes: 6 additions & 0 deletions readmes/README_en.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,12 @@ After deployment, set the following variables:
| `DB_DRIVER` | `auto` | `auto` / `blob` / `cfkv` / `kv` / `d1` / `do` / `mysql` |
| `DB_CIPHER` | `none` | At-rest cipher for sensitive fields, e.g. `aes-256-gcm` |

> **KV quota note**: `DB_FORMAT=key` maintains a global index key (`openlist_key_index`),
> so after the first read of legacy data, load/save no longer consume KV `list` quota.
> Each save still writes one entry per record (about 100+ with the default config), and
> the Cloudflare KV free tier allows only 1000 writes/day — prefer `DB_DRIVER=d1` or
> `DB_FORMAT=map` for long-term use.

For the full variable list, see [Configuration variables](https://doc.oplist.org/ecosystem/official_worker/guide_env).

## Support
Expand Down
176 changes: 100 additions & 76 deletions scripts/test-task-api.mts
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
// Quick smoke test for task management API endpoints
import app from "../src/backend/index"
// Smoke test for the task management API.
//
// 契约对齐 Go 版(server/handles/task.go):
// - 列表/清理:200
// - 单个任务操作(info/cancel/delete/retry):tid 不存在 → 404(防探测)
// - 批量操作:200,返回 {tid: error} 映射
import { Hono } from "hono"
import { saveDb } from "../src/backend/internal/model/db"
import { setupRouter } from "../src/backend/server/router"

// 不用 src/backend/index 的默认导出:它带「存储配置错误 → 全局 503」拦截,
// 在无绑定的本地冒烟环境里会把所有 API 直接拦掉。这里只挂路由。
const api = new Hono()
setupRouter(api)
const app = new Hono()
app.route("/api", api)

const TOKEN = "TASK_API_SMOKE_TOKEN"
const env: any = { JWT_SECRET: "TASK_API_SMOKE_CRON_SECRET" }
const headers = { Authorization: `Bearer ${TOKEN}` }

async function test(name: string, fn: () => Promise<void>) {
try {
Expand All @@ -12,101 +30,107 @@ async function test(name: string, fn: () => Promise<void>) {
}

async function req(method: string, path: string, body?: any) {
const res = await app.request(path, {
method,
headers: body !== undefined ? { "Content-Type": "application/json" } : {},
body: body !== undefined ? JSON.stringify(body) : undefined,
})
const res = await app.request(
path,
{
method,
headers:
body !== undefined
? { ...headers, "Content-Type": "application/json" }
: headers,
body: body !== undefined ? JSON.stringify(body) : undefined,
},
env,
)
const json = await res.json().catch(() => null)
return { status: res.status, json }
}

const types = ["upload", "copy", "move", "offline_download"]

await test("GET /api/task/upload/undone returns task list", async () => {
const { status, json } = await req("GET", "/api/task/upload/undone")
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
if (!Array.isArray(json.data)) throw new Error("data is not an array")
})

await test("GET /api/task/upload/done returns task list", async () => {
const { status, json } = await req("GET", "/api/task/upload/done")
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
if (!Array.isArray(json.data)) throw new Error("data is not an array")
})

await test("GET /api/task/{all types}/{done} return 200", async () => {
await saveDb(
{
settings: [{ key: "token", value: TOKEN }],
users: [
{
id: 1,
username: "admin",
password: "xxx",
role: 2,
permission: 0,
base_path: "/",
disabled: false,
},
],
storages: [],
shares: [],
},
env,
)

const types = [
"upload",
"copy",
"move",
"offline_download",
"decompress",
"seed_generate",
"seed_save",
]

await test("GET /api/task/{all types}/{undone|done} returns TaskInfo[]", async () => {
for (const type of types) {
for (const done of ["undone", "done"]) {
const { status } = await req("GET", `/api/task/${type}/${done}`)
if (status !== 200) throw new Error(`${type}/${done}: status ${status}`)
for (const state of ["undone", "done"]) {
const { status, json } = await req("GET", `/api/task/${type}/${state}`)
if (status !== 200) throw new Error(`${type}/${state}: status ${status}`)
if (json.code !== 200) throw new Error(`${type}/${state}: code ${json.code}`)
if (!Array.isArray(json.data)) {
throw new Error(`${type}/${state}: data is not an array`)
}
}
}
})

await test("POST /api/task/upload/clear_done works", async () => {
const { status, json } = await req("POST", "/api/task/upload/clear_done")
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
await test("unknown task type returns 404", async () => {
const { status } = await req("GET", "/api/task/not-a-type/undone")
if (status !== 404) throw new Error(`status ${status}`)
})

await test("POST /api/task/upload/clear_succeeded works", async () => {
const { status, json } = await req("POST", "/api/task/upload/clear_succeeded")
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
})

await test("POST /api/task/upload/retry_failed works", async () => {
const { status, json } = await req("POST", "/api/task/upload/retry_failed")
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
})

await test("POST /api/task/upload/retry?tid= works", async () => {
const { status, json } = await req("POST", "/api/task/upload/retry?tid=1")
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
})

await test("POST /api/task/upload/cancel?tid= works", async () => {
const { status, json } = await req("POST", "/api/task/upload/cancel?tid=1")
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
await test("clear_done / clear_succeeded work", async () => {
for (const action of ["clear_done", "clear_succeeded"]) {
const { status, json } = await req("POST", `/api/task/copy/${action}`)
if (status !== 200) throw new Error(`${action}: status ${status}`)
if (json.code !== 200) throw new Error(`${action}: code ${json.code}`)
}
})

await test("POST /api/task/upload/delete?tid= works", async () => {
const { status, json } = await req("POST", "/api/task/upload/delete?tid=1")
await test("retry_failed works", async () => {
const { status, json } = await req("POST", "/api/task/copy/retry_failed")
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
})

await test("POST /api/task/upload/cancel_some works", async () => {
const { status, json } = await req("POST", "/api/task/upload/cancel_some", [
"1",
"2",
])
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
await test("single-task operations return 404 for a missing tid", async () => {
for (const action of ["info", "cancel", "delete", "retry"]) {
const { status } = await req("POST", `/api/task/copy/${action}?tid=missing`)
if (status !== 404) throw new Error(`${action}: status ${status}`)
}
})

await test("POST /api/task/upload/delete_some works", async () => {
const { status, json } = await req("POST", "/api/task/upload/delete_some", [
"1",
"2",
])
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
await test("bulk operations report per-tid errors", async () => {
for (const action of ["cancel_some", "delete_some", "retry_some"]) {
const { status, json } = await req("POST", `/api/task/copy/${action}`, [
"missing-1",
"missing-2",
])
if (status !== 200) throw new Error(`${action}: status ${status}`)
if (json.data?.["missing-1"] !== "task not found") {
throw new Error(`${action}: unexpected payload ${JSON.stringify(json.data)}`)
}
}
})

await test("POST /api/task/upload/retry_some works", async () => {
const { status, json } = await req("POST", "/api/task/upload/retry_some", [
"1",
"2",
])
if (status !== 200) throw new Error(`status ${status}`)
if (json.code !== 200) throw new Error(`code ${json.code}`)
await test("unauthenticated task API returns 401", async () => {
const res = await app.request("/api/task/copy/undone", {}, env)
if (res.status !== 401) throw new Error(`status ${res.status}`)
})

console.log("\nAll task API tests completed.")
Loading
Loading