downloader

ClipFetch Backend

Django + Django REST Framework backend for ClipFetch, a video/audio downloader web application. The frontend is developed separately – this repository contains the backend only: metadata extraction, format selection, downloading, merging, audio conversion, access control and clean-up.

No frontend files are part of this repository. The API contract below is the interface your friend needs to implement against.

Architecture at a glance

                    ┌──────────────────────────────────────────────┐
  POST /api/analyze │  views.py        validate the request         │
  POST /api/download│      ↓                                       │
  GET  /api/files/… │  serializers.py → services.py (the logic)    │
                    │      ↓                        ↓               │
                    │  validators.py          cache.py  (Redis)     │
                    │  (URL + SSRF)                ↓               │
                    │      ↓              fast_*.py → extractor.py  │
                    │  policy.py             (yt-dlp / fast paths) │
                    │  (every request)              ↓               │
                    │      ↓                   ffmpeg.py           │
                    │   the platform            (merge/convert)    │
                    └──────────────────────────────────────────────┘
                                      ↓
                              media/downloads/<file>

The request flow (one request, one path):

views.py validates the body with serializers.py.
validators.py checks the URL: scheme, host allow-list, port, DNS resolution (SSRF protection).
services.py asks cache.py first – on a hit the answer is returned without touching the network.
On a miss, one of the fast paths (fast_youtube.py, fast_tiktok.py, fast_facebook.py, fast_instagram.py) extracts the metadata; if that fails, extractor.py (yt-dlp) does it.
The download uses the direct CDN url when the fast path provided one, otherwise yt-dlp downloads the stream and ffmpeg.py merges the video with the audio track.
The file is stored in media/downloads/, the job in the database, and the frontend receives a time limited link.

The three rules that matter when changing this code:

Rule	Why
The extraction logic never lives in a view	The views are reused by the API and the Celery worker
Every fast path must fall back to yt-dlp	A platform change can then never break the API
The signed CDN url never leaves the server	It is stripped from the API response (`extractor.public_formats`)

Architecture at a glance
Features
Verified platform support
Requirements
Installation
Configuration
Running the development server
API reference
Caching & performance
Frontend integration guide
Project structure
Security
Tests
Background jobs (Celery)
Clean-up
Production deployment
Troubleshooting
Legal notice

Handoff: what to do first

If you (or an AI agent) are taking over this repository, this is the order that works:

cp .env.example .env and set DJANGO_SECRET_KEY and CORS_ALLOWED_ORIGINS.
pip install -r requirements.txt, python manage.py migrate, python manage.py runserver → check http://127.0.0.1:8000/api/health/.
Read Architecture at a glance and the API reference – that is the whole contract.
Run the tests: python manage.py test (184 tests, no network needed except the two opt-in integration tests).
Connect the frontend: set its origin in CORS_ALLOWED_ORIGINS and use POST /api/analyze/ + POST /api/download/.
Before going live: work through the production deployment checklist.

Where to change what

Task	File
A new endpoint	`downloader/urls.py` + `downloader/views.py`
The request/response shape	`downloader/serializers.py` / `downloader/views.py`
The extraction logic	`downloader/extractor.py` + `downloader/fast_*.py`
The download/merge logic	`downloader/services.py` + `downloader/ffmpeg.py`
The URL/SSRF rules	`downloader/validators.py` + `downloader/policy.py`
A new setting	`config/settings.py` and `.env.example` and the README table
The cache behaviour	`downloader/cache.py`

Features

Area	What is implemented
Analysis	Real metadata via `yt-dlp` (title, thumbnail, duration, uploader, real available formats and resolutions) – nothing is hard coded
Formats	Every real format is returned with `format_id`, resolution label, container, codecs, file size and audio/video flags
Video download	Progressive formats are downloaded directly, video-only formats are merged with the best audio track via FFmpeg
Containers	The result is an MP4 whenever possible (the video is copied, only a non-MP4 audio track such as Opus is transcoded to AAC); Matroska is used as fallback
Audio download	`mp3`, `m4a` and `wav` (WAV only when FFmpeg is available)
Safe files	Sanitised, ASCII-only file names; no path traversal, no user input in shell commands
Access control	Every download gets a random, time limited token; links expire and are removed automatically
Limits	Max file size, max duration, socket/extraction/download timeouts, per-IP rate limits, global download concurrency limit
SSRF protection	Host allow-list, DNS resolution checks, redirect validation on every request made by yt-dlp
Clean-up	Expired files, old job rows and stale temporary files are removed automatically
Caching	Metadata, finished downloads and DNS lookups are cached – repeated requests are answered in milliseconds instead of seconds
Production	PostgreSQL, Redis, Celery, gunicorn + nginx, structured and redacted logging

Verified platform support

The platforms were tested against the live sites (yt-dlp 2026.10.21, no cookies):

Platform	Status	Notes
YouTube	✅ works	Real extraction, 20 formats up to 4K. A fast path (`pytubefix`) makes the metadata extraction ~2.4× faster than yt-dlp.
TikTok	✅ works	Public videos are extracted and downloaded. A fast path (the JSON embedded in the video page) makes the metadata ~1.7× faster.
Facebook	✅ works (partially)	Public page videos work. A fast path (the JSON embedded in the page) makes the metadata ~1.5× faster and delivers the direct CDN url.
Instagram	✅ works	Public posts and reels are extracted without a login via the GraphQL fast path (~3.6× faster than yt-dlp). Private accounts still need your own cookies.

The backend never circumvents DRM, paywalls, access controls or rate limits. When a platform requires a login, the API answers with a clear media_unavailable error (or, if you provide your own cookies, with the media).

Requirements

Python 3.10+ (developed and tested with 3.12)
FFmpeg (required for merging video+audio and for converting audio)
Optional: Redis (rate limiting across processes + background jobs), PostgreSQL (production database), Deno (recommended for YouTube)

FFmpeg installation

Debian / Ubuntu / Mint

sudo apt update
sudo apt install -y ffmpeg
ffmpeg -version

macOS

brew install ffmpeg

Windows (PowerShell)

winget install Gyan.FFmpeg
# or: choco install ffmpeg

Docker

RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg && rm -rf /var/lib/apt/lists/*

Without a system install (optional, handy for development)

pip install imageio-ffmpeg   # ships a static ffmpeg binary

The backend looks for FFmpeg in this order: CLIPFETCH_FFMPEG_PATH → PATH → imageio-ffmpeg. GET /api/health/ reports whether FFmpeg is available.

Installation

git clone <your-repo> website-backend
cd website-backend

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

pip install -r requirements.txt

cp .env.example .env               # adjust the values (see below)
python manage.py migrate
python manage.py runserver

The API is now available at http://127.0.0.1:8000/api/.

If python3 -m venv fails with “ensurepip is not available” install the venv package first: sudo apt install python3-venv (or sudo apt install python3.12-venv).

Configuration

All settings live in .env (see .env.example for the copy‑paste ready file). Everything below is read from the environment – nothing is hard coded.

Django

Variable	Default	Description
`DJANGO_SECRET_KEY`	insecure dev key	Must be changed for production
`DJANGO_DEBUG`	`True`	Set to `False` in production
`DJANGO_ALLOWED_HOSTS`	`localhost,127.0.0.1`	Comma separated hostnames
`DJANGO_CSRF_TRUSTED_ORIGINS`	–	Only needed if session based views are added later
`DJANGO_NUM_PROXIES`	`0`	Number of reverse proxies (correct client IP for the rate limits)
`DJANGO_LOG_LEVEL`	`INFO`	Log level of the application
`DJANGO_SECURE_SSL_REDIRECT`	`False`	Enable in production (TLS is terminated by the proxy)
`DJANGO_SECURE_HSTS_SECONDS`	`0`	e.g. `31536000` in production
`DJANGO_SECURE_HSTS_INCLUDE_SUBDOMAINS`	`False`	Only with HSTS
`DJANGO_SECURE_HSTS_PRELOAD`	`False`	Only with HSTS
`DJANGO_TRUST_PROXY_SSL_HEADER`	`False`	Trust `X-Forwarded-Proto` of the reverse proxy
`DJANGO_SESSION_COOKIE_SECURE`	`True` (prod)	Only relevant with session views
`DJANGO_CSRF_COOKIE_SECURE`	`True` (prod)	Only relevant with session views

CORS (the frontend)

Variable	Default	Description
`CORS_ALLOWED_ORIGINS`	`http://localhost:5173,…`	The frontend origin(s) – comma separated
`CORS_ALLOW_CREDENTIALS`	`False`	Keep `False` for the token-less API
`CORS_ALLOW_ALL_ORIGINS`	`False`	Never enable in production

Database, cache and broker

Variable	Default	Description
`POSTGRES_DB` / `POSTGRES_USER`	–	When both are set, PostgreSQL is used instead of SQLite
`POSTGRES_PASSWORD`	–
`POSTGRES_HOST` / `POSTGRES_PORT`	`localhost` / `5432`
`POSTGRES_CONN_MAX_AGE`	`60`	Keep the connection open (seconds)
`REDIS_URL`	–	e.g. `redis://localhost:6379/0` – required for a shared cache with more than one worker
`CELERY_BROKER_URL`	`REDIS_URL` or `memory://`
`CELERY_RESULT_BACKEND`	`REDIS_URL` or `cache+memory://`

Media extraction

Variable	Default	Description
`CLIPFETCH_MAX_DOWNLOAD_SIZE`	`4294967296` (4 GB)	Maximum file size
`CLIPFETCH_MAX_DURATION`	`7200` (2 h)	Maximum media duration
`CLIPFETCH_MAX_REDIRECTS`	`5`	Maximum redirect hops
`CLIPFETCH_SOCKET_TIMEOUT`	`20`	Network socket timeout (s)
`CLIPFETCH_EXTRACT_TIMEOUT`	`45`	Metadata extraction timeout (s)
`CLIPFETCH_DOWNLOAD_TIMEOUT`	`600`	Full download timeout (s)
`CLIPFETCH_ALLOW_HTTP`	`False`	Allow plain `http://` links (https only by default)
`CLIPFETCH_VERIFY_TLS`	`True`	Do not disable
`CLIPFETCH_USER_AGENT`	Chrome UA	User agent of the extraction requests
`CLIPFETCH_MAX_CONCURRENT_DOWNLOADS`	`3`	Parallel downloads per server
`CLIPFETCH_CONCURRENT_FRAGMENT_DOWNLOADS`	`1`	Parallel fragment downloads (HLS/DASH)
`CLIPFETCH_YTDLP_CHECK_FORMATS`	`auto`	`auto` (safe) · `false` (fastest) · `selected` (verify all)
`CLIPFETCH_YTDLP_PLAYER_CLIENT`	–	e.g. `android_vr,ios` – faster, but fewer formats
`CLIPFETCH_YTDLP_IMPERSONATE`	–	e.g. `chrome` – needs `pip install curl-cffi`
`CLIPFETCH_YTDLP_JS_RUNTIME`	–	e.g. `deno` – needed by YouTube for the full format list
`CLIPFETCH_YTDLP_COOKIES_FILE`	–	Your own cookie file (Netscape format) for platforms that need a login
`CLIPFETCH_YTDLP_COOKIES_FROM_BROWSER`	–	e.g. `chrome` or `firefox:Profile 1`

Fast extraction paths (all optional, all with a yt-dlp fallback)

Variable	Default	Description
`CLIPFETCH_FAST_YOUTUBE`	`True`	`pytubefix` (InnerTube API) – ~2.4× faster
`CLIPFETCH_FAST_TIKTOK`	`True`	The JSON embedded in the page – ~1.7× faster
`CLIPFETCH_FAST_FACEBOOK`	`True`	The JSON embedded in the page – ~1.5× faster
`CLIPFETCH_FAST_INSTAGRAM`	`True`	The public GraphQL API – ~3.6× faster, no login needed
`CLIPFETCH_FAST_TIMEOUT`	`8`	Hard timeout of the fast paths (s)

Caching, jobs and storage

Variable	Default	Description
`CLIPFETCH_ANALYSIS_CACHE_TTL`	`3600`	How long extracted metadata is reused (s, `0` = off)
`CLIPFETCH_DOWNLOAD_CACHE_TTL`	`1800`	How long a finished download is reused for the same url + format (s)
`CLIPFETCH_DNS_CACHE_TTL`	`300`	How long a DNS resolution is reused (s)
`CLIPFETCH_AUTO_CLEANUP`	`True`	Run the clean-up during the API requests
`CLIPFETCH_CLEANUP_INTERVAL`	`300`	Seconds between clean-up runs
`CLIPFETCH_TMP_MAX_AGE`	`3600`	Age after which abandoned temp files are removed (s)
`CLIPFETCH_JOB_RETENTION_DAYS`	`7`	Age after which finished job rows are deleted
`CLIPFETCH_LINK_TTL`	`3600`	Validity of a temporary download link (s)
`CLIPFETCH_RATE_ANALYZE`	`60/hour`	Per-IP analysis rate limit
`CLIPFETCH_RATE_DOWNLOAD`	`20/hour`	Per-IP download rate limit
`CLIPFETCH_RATE_FILE`	`240/hour`	Per-IP file download rate limit
`CLIPFETCH_ASYNC`	`False`	`True` → downloads run as Celery jobs (`202` + polling)
`CLIPFETCH_DOWNLOAD_DIR`	`media/downloads`	Where finished files are stored
`CLIPFETCH_TMP_DIR`	`media/tmp`	Intermediate files

FFmpeg

Variable	Default	Description
`CLIPFETCH_FFMPEG_PATH`	auto	Explicit FFmpeg binary (auto: `PATH` → `imageio-ffmpeg`)
`CLIPFETCH_FFPROBE_PATH`	auto	Explicit ffprobe binary (optional)

Gunicorn / Docker

Variable	Default	Description
`WEB_PORT`	`8000`	Published port of the compose stack
`GUNICORN_BIND`	`0.0.0.0:8000`
`GUNICORN_WORKERS`	`min(4, cpu_count)`	Worker processes
`GUNICORN_THREADS`	`4`	Threads per worker
`GUNICORN_TIMEOUT`	`900`	Must be > `CLIPFETCH_DOWNLOAD_TIMEOUT`
`GUNICORN_MAX_REQUESTS`	`2000`	Worker recycling
`GUNICORN_PRELOAD`	`1`	Preload the app (faster start)

Running the development server

source .venv/bin/activate
python manage.py migrate
python manage.py runserver 0.0.0.0:8000

Useful commands:

python manage.py test                          # run the test suite
python manage.py cleanup_downloads             # remove expired downloads now
python manage.py check                          # Django configuration check
celery -A config worker -l info                # background worker (async mode)
celery -A config beat -l info                   # periodic clean-up (async mode)

API reference

All endpoints speak JSON. Every response uses the same envelope:

// success
{ "success": true,  "...": "...", "error": null }

// failure
{ "success": false, "error": { "code": "unsupported_url", "message": "…" } }

The responses of /api/analyze/ and /api/download/ additionally contain the header X-Cache: HIT (served from the cache) or X-Cache: MISS (freshly computed) – see Caching & performance.

Endpoint contract

Method	Path	Auth	Rate limit	Success	Failure
`POST`	`/api/analyze/`	none	60/hour per IP	`200` + metadata + formats	`400` / `502` / `504`
`POST`	`/api/download/`	none	20/hour per IP	`200` + job + `download_url` (or `202` in async mode)	`400` / `413` / `429` / `502` / `504`
`GET`	`/api/files/<job_id>/<token>/`	token in the URL	240/hour per IP	`200` + the file	`404` (wrong token) / `410` (expired)
`GET`	`/api/jobs/<job_id>/<token>/`	token in the URL	240/hour per IP	`200` + the job state	`404` / `410`
`GET`	`/api/health/`	none	none	`200` + `{status, features}`	–

Authentication: the API is public and stateless – there is no login, no API key and no CSRF token. The download links are protected by a random 256-bit token that expires.
Content type: always application/json (the CORS preflight is handled automatically for the allowed origins).
Errors: every error has the same shape, so one error handler in the frontend is enough.

`POST /api/analyze/`

Extracts the real metadata of a media link. Nothing is downloaded – only the metadata is fetched (skip_download).

Request

{ "url": "https://www.youtube.com/watch?v=aqz-KE-bpKQ" }

Response 200

{
  "success": true,
  "platform": "youtube",
  "title": "Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film",
  "thumbnail": "https://i.ytimg.com/vi_webp/aqz-KE-bpKQ/maxresdefault.webp",
  "duration": 635,
  "extractor": "Youtube",
  "uploader": "Blender",
  "url": "https://www.youtube.com/watch?v=aqz-KE-bpKQ",
  "is_live": false,
  "available_formats": [
    {
      "format_id": "299",
      "height": 1080,
      "width": 1920,
      "ext": "mp4",
      "resolution": "1080p",
      "dimensions": "1920x1080",
      "has_audio": false,
      "has_video": true,
      "is_progressive": false,
      "vcodec": "avc1.64002a",
      "acodec": null,
      "fps": 60.0,
      "tbr": 3247.821,
      "filesize": 257619653,
      "filesize_approx": 257619597,
      "estimated_filesize": 257619653,
      "too_large": false,
      "protocol": "https",
      "note": "1080p60"
    }
  ],
  "available_resolutions": ["4K", "1440p", "1080p", "720p", "480p", "360p", "240p", "144p"],
  "error": null
}

available_formats contains only formats that really exist for this media.
resolution is a human readable label (240p … 1440p, 4K). For vertical videos the shorter side is used, so a 1080×1920 clip is reported as 1080p.
has_audio / has_video tell the frontend whether the format already contains an audio track (progressive) or has to be merged with one.
filesize may be null when the platform does not announce the size – estimated_filesize is then calculated from the bitrate and duration.
too_large is true when the format exceeds CLIPFETCH_MAX_DOWNLOAD_SIZE. The frontend should disable those options; requesting one anyway returns file_too_large (HTTP 413).

`POST /api/download/`

Downloads a video or an audio track and returns a temporary, access controlled download link.

Request

{
  "url": "https://www.youtube.com/watch?v=aqz-KE-bpKQ",
  "format_id": "299",        // optional – a format id from /api/analyze/
  "media_type": "video",     // "video" (default) | "audio"
  "audio_format": "mp3"      // only for media_type="audio": mp3 | m4a | wav
}

Response 200 (finished)

{
  "success": true,
  "job_id": "42b8aa9b-a81f-4948-b548-53d4b803a930",
  "status": "completed",
  "platform": "youtube",
  "title": "Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film",
  "media_type": "video",
  "audio_format": null,
  "format_id": "299",
  "filename": "Big-Buck-Bunny-60fps-4K-Official-Blender-Foundation-Short-Film.mkv",
  "size_bytes": 28507874,
  "mime_type": "video/x-matroska",
  "duration": 635.0,
  "created_at": "2026-10-22T06:11:00.123456+00:00",
  "completed_at": "2026-10-22T06:11:32.654321+00:00",
  "expires_at": "2026-10-22T07:11:32.654321+00:00",
  "error": null,
  "download_url": "/api/files/42b8aa9b-a81f-4948-b548-53d4b803a930/KtZiolrBK04ywGS0iXRDAX7uZ-RxAO8n36RNPARzU20/",
  "status_url": "/api/jobs/42b8aa9b-a81f-4948-b548-53d4b803a930/KtZiolrBK04ywGS0iXRDAX7uZ-RxAO8n36RNPARzU20/"
}

Response 202 (background mode, CLIPFETCH_ASYNC=1)

{
  "success": true,
  "job_id": "…",
  "status": "queued",
  "status_url": "/api/jobs/<job_id>/<token>/",
  "message": "The download has been queued.",
  "error": null
}

Poll status_url until status becomes completed (then download_url is present) or failed (then error contains code and message).

Audio downloads

media_type: "audio" + audio_format: "mp3" | "m4a" | "wav".
format_id may be omitted – the best available audio track is used.
If format_id is given it must exist and contain an audio stream, otherwise the API answers with format_not_available.
wav requires FFmpeg; without FFmpeg the API answers with ffmpeg_unavailable.

`GET /api/files/<job_id>/<token>/`

Streams the finished file (Content-Disposition: attachment). The token is random, per job and expires after CLIPFETCH_LINK_TTL seconds.

wrong token → 404 not_found
expired link → 410 link_expired
file already deleted → 404 not_found

`GET /api/jobs/<job_id>/<token>/`

Returns the current state of a job (same token as the download link).

`GET /api/health/`

{
  "status": "ok",
  "service": "clipfetch-backend",
  "time": "2026-10-22T06:13:21.380643+00:00",
  "features": { "ffmpeg": true, "async_jobs": false }
}

Error codes

`error.code`	HTTP	Meaning
`validation_error`	400	The request body is invalid (missing/invalid field)
`invalid_url`	400	The URL is malformed, not http(s), contains credentials or uses an unusual port
`unsupported_url`	400	The platform is not supported
`blocked_url`	400	The URL points at a private/internal address (SSRF protection)
`invalid_request`	400	The request could not be processed
`format_not_available`	400	The requested `format_id` does not exist / has no audio
`unsupported_media`	400	DRM protected media, live stream
`media_unavailable`	404	Private, removed, region locked or login required
`not_found`	404	Unknown job or wrong token
`link_expired`	410	The temporary download link expired
`file_too_large`	413	The file exceeds `CLIPFETCH_MAX_DOWNLOAD_SIZE`
`duration_too_long`	413	The media exceeds `CLIPFETCH_MAX_DURATION`
`extraction_failed`	502	yt-dlp could not extract the media
`download_failed`	502	The file itself could not be downloaded (network/stream error)
`ffmpeg_error`	500	Merging/conversion failed
`ffmpeg_unavailable`	503	FFmpeg is not installed on the server
`upstream_error`	503	The platform rate limited the request
`rate_limited`	429	Too many requests (see `Retry-After`)
`too_many_downloads`	429	The server is at its download concurrency limit
`timeout`	504	The extraction/download took too long

Caching & performance

The API is designed so that the expensive work (talking to YouTube/TikTok/…) happens once. Everything that can be reused is cached through the Django cache framework, which uses the local memory cache in development and Redis in production (REDIS_URL).

Cache	Key	Default TTL	What it saves
`analysis`	`clipfetch:analysis:<sha256(url)>`	1 h	The whole yt-dlp metadata extraction (2–4 s)
`download`	`clipfetch:download:<sha256(url+format)>`	30 min	The complete file download (10 s – minutes)
`dns`	`clipfetch:dns:<hostname>`	5 min	A DNS lookup per request
throttle	`throttle_<scope>_<ip>`	1 h	DRF’s built-in rate limiter state
concurrency	`clipfetch:concurrent-downloads`	1 h	The counter of parallel downloads

Measured on a real YouTube video (curl -w "%{time_total}"):

Request	Cold	Warm (cache hit)	Speed-up
`POST /api/analyze/`	2.32 s	0.0045 s	~515×
`POST /api/download/`	33.7 s	0.071 s	~474×

Every response carries an X-Cache: HIT|MISS header so you can see whether the answer came from the cache.

How it works

/api/analyze/ returns the cached metadata for a URL that was analysed before.
/api/download/ reuses an already downloaded file when the same URL + format + media type is requested again and the file is still available (the file is shared between the jobs, the clean-up removes it only when no job needs it anymore).
The download path no longer extracts the metadata a second time – it uses the cached analysis to validate the requested format, so only one yt-dlp extraction is left (the one inside the actual download).
A download also warms the analysis cache, so a later /api/analyze/ call is instant.
The DNS cache removes a DNS lookup from every request.

Using Redis (recommended for production)

# install & start
sudo apt install redis-server && sudo systemctl start redis-server

# .env
REDIS_URL=redis://127.0.0.1:6379/0

With Redis the cache is shared between all gunicorn workers and all servers, so the rate limits and the download concurrency limit are enforced globally instead of per process.

Cache failures never break the API – every cache operation is wrapped in a guard: if Redis is down the API keeps working, just without the cache.

Where the time actually goes

Measured on a real YouTube video (curl -w "%{time_total}"):

Phase	Cold	Warm (cache hit)	Controlled by
`POST /api/analyze/`	1.0 – 2.5 s	0.004 s	the extractor (see below)
`POST /api/download/`	= transfer time	0.05 s	the network/CDN, not yt-dlp

So after the cache, the remaining cost is:

the extraction (~1–2.5 s) – fetching the watch page, the player script and solving the signature/PO-token challenges.
the byte transfer – the CDN download. Benchmarked at 0.4–1.3 MB/s from this machine, i.e. the download time is file size / bandwidth. No extractor option can change that.

Alternatives to yt-dlp (measured)

yt-dlp is not the only option. Three fast paths are built in – they talk to the platform directly and skip the parts of yt-dlp that are slow:

Platform	Fast path	Cold metadata	vs yt-dlp	Why it is faster
YouTube	`pytubefix` (InnerTube API)	0.74 – 1.09 s	~2.4×	yt-dlp has to download and interpret the player JavaScript (and solve the PO-token challenges); the InnerTube API does not need it
TikTok	the JSON embedded in the video page	0.52 – 0.83 s	~1.7×	TikTok puts the complete media information into the page as JSON – yt-dlp needs an extra API round trip for it
Facebook	the JSON embedded in the video page (browser headers)	0.95 – 1.16 s	~1.5×	with the `Sec-Fetch-` headers Facebook returns the complete video data in one* request – title, duration, dimensions and the direct CDN urls
Instagram	the public GraphQL API (`xdt_api__v1__media__shortcode`)	0.45 – 0.52 s	~3.6×	the web page is behind the login wall, the GraphQL endpoint is not – it returns the direct CDN urls, the thumbnail, the author and even the duration (in the inline DASH manifest) without a login

How the fast paths work

downloader/fast_youtube.py extracts the metadata with pytubefix and converts it into the same format structure as yt-dlp, so the rest of the code does not change.
downloader/fast_tiktok.py reads the __UNIVERSAL_DATA_FOR_REHYDRATION__ JSON of the video page and rebuilds the formats – including the same format ids that yt-dlp generates (h264_540p_2240963), so the frontend can pick a format from the fast list and download it with yt-dlp.
downloader/fast_facebook.py fetches the page with browser headers and parses the embedded video object. It returns the direct CDN url, so the download does not need an extraction at all any more.
downloader/fast_instagram.py queries the public GraphQL endpoint (doc_id=27128499623469141, the same query that instaloader uses) and reads the direct CDN urls, the thumbnail, the author, the dimensions and the duration (from the inline DASH manifest). It works without a login – unlike the normal Instagram web page that yt-dlp has to parse.
The format ids are the YouTube itags / TikTok gear ids in both cases, so the frontend never sees a difference. Downloads that use a base id (without the CDN mirror suffix) are resolved with a prefix match (best[format_id^=…]).
The signed CDN url is never returned to the frontend – it is only kept in the server side cache and used for the download.
Every error falls back to yt-dlp automatically – if the fast path is not installed, is disabled or the platform changed, the request is served by yt-dlp. The API can never break because of it.
The actual download still uses yt-dlp (it handles HLS/DASH, retries, range requests and the merging robustly) – except for the direct urls of the fast paths, which are downloaded as they are.
CLIPFETCH_FAST_YOUTUBE=False / CLIPFETCH_FAST_TIKTOK=False / CLIPFETCH_FAST_FACEBOOK=False / CLIPFETCH_FAST_INSTAGRAM=False disable them.

The repositories that were evaluated

Repository	Result
`pytubefix` (11.2.1, MIT, actively maintained)	✅ used for the YouTube fast path – 2.4× faster than yt-dlp
`gallery-dl` (1.32.17, 8.9k commits, 1000+ sites)	✅ the technique was copied (browser headers + the embedded JSON) into `fast_facebook.py` – but the library itself is not needed, so it is not a dependency. Measured: 1.5× faster for Facebook, but 10.4 s for Instagram (login wall)
`instaloader` (4.15.4, MIT, actively maintained)	✅ its GraphQL query was used for the Instagram fast path (`doc_id=27128499623469141`) – the query itself is only ~40 lines, so the library is not needed as a dependency. As a library it was slower (3.34 s) and returned no video data without a login
`facebook-scraper` (0.2.59)	❌ rejected – abandoned since 2022, it has no API for a single video url (it scans page posts), and it pulls in `pyppeteer` which downgrades `urllib3` and breaks yt-dlp
`socialapis-sdk`	❌ rejected – a hosted commercial API (needs a token, 200 free calls/month), not self-hosted
Cobalt (self-hostable)	❌ rejected – a whole extra Node service with its own API and an AGPL licence
Hand-written InnerTube client	❌ rejected – needs the player JavaScript for the signature cipher, breaks on every YouTube change

What was optimised

Change	Effect
Metadata + download cache	2.4 s → 0.004 s / 33 s → 0.05 s (~500×)
Fast YouTube path (`pytubefix`)	cold analyze 2.45 s → 1.01 s (~2.4×)
Fast TikTok path (page JSON)	cold analyze 1.30 s → 0.76 s (~1.7×)
Fast Facebook path (page JSON)	cold analyze 1.46 s → 1.01 s (~1.5×)
Fast Instagram path (GraphQL)	cold analyze 1.82 s → 0.50 s (~3.6×) and it works without a login
Direct url download (Facebook)	the download skips the extraction completely (−1.4 s)
No second extraction in the download path	−2 s per download
Format de-duplication (53 → 20 formats, prefers HTTPS + h264)	response 17.4 KB → 7.4 KB (−57 %)
GZip compression	7.4 KB → ~2.2 KB over the wire
`CLIPFETCH_YTDLP_CHECK_FORMATS=false` (opt-in)	cold analyze 2.45 s → 2.03 s (−17 %)

What was measured but deliberately not changed

Honest results of the benchmarks – these options exist, but the default stays conservative:

Option	Measured	Decision
`concurrent_fragment_downloads` 1 → 8	9.05 s vs 4.77 s … but 8.87 s vs 6.86 s in the reversed run – pure network noise (0.37–1.3 MB/s)	left at `1` (server friendly), configurable
`curl_cffi` browser impersonation	1.92 s vs 2.01 s – no speed difference (it mainly avoids the “confirm you are not a bot” challenge)	available as `CLIPFETCH_YTDLP_IMPERSONATE` for reliability
`player_client=android_vr`	1.60 s instead of 1.96 s, but only 5 formats instead of 53	available as `CLIPFETCH_YTDLP_PLAYER_CLIENT` (speed vs completeness trade-off)

Bottom line: the API is only as fast as yt-dlp and the platform. The only way to be faster than that is to not do the work twice (the cache) – which is exactly what this implementation does. Everything else is a trade-off between speed and the number of available formats.

Frontend integration guide

Base URL – point the frontend at the backend, e.g. const API = process.env.VITE_API_URL ?? "http://localhost:8000";
CORS – add the frontend origin to CORS_ALLOWED_ORIGINS in the backend .env and restart the backend. No credentials are needed (CORS_ALLOW_CREDENTIALS=False), so fetch works without any extra setup.
CSRF – the API is stateless and token-less, therefore no CSRF token is required. Do not send cookies. If you later add session based endpoints, add the frontend origin to DJANGO_CSRF_TRUSTED_ORIGINS as well.
Typical flow

// 1. analyse
const analyze = await fetch(`${API}/api/analyze/`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ url }),
});
const info = await analyze.json();
if (!info.success) throw new Error(info.error.message);

// 2. download
const dl = await fetch(`${API}/api/download/`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ url, format_id: chosenFormatId, media_type: "video" }),
});
const job = await dl.json();
if (!job.success) throw new Error(job.error.message);

// 3. save the file (the URL is relative to the API base)
window.location.href = `${API}${job.download_url}`;

Polling (async mode) – poll job.status_url every 2–3 seconds until status is completed or failed.
Error handling – always check success and show error.message (error.code can be used for localisation).
The download_url is relative – prefix it with the API base URL. It contains a random token and expires after CLIPFETCH_LINK_TTL seconds, so request a new download instead of caching the link.

Integration checklist

CORS_ALLOWED_ORIGINS in .env contains the exact frontend origin (e.g. https://app.example.com) and the backend was restarted
the frontend sends Content-Type: application/json (the preflight is handled automatically)
the frontend reads available_formats[].format_id and sends it back as format_id – it is not the same as an index
the frontend treats download_url as a relative path and prefixes the API base URL
the frontend shows error.message (all errors are HTTP 4xx/5xx with the same JSON envelope)
the frontend does not need a CSRF token (the API is stateless)
in production: DJANGO_DEBUG=False, a real DJANGO_SECRET_KEY, PostgreSQL + Redis, CLIPFETCH_ASYNC=True with a Celery worker, and a clean-up cron/timer

Project structure

website-backend/
├── config/                     # Django project (settings, urls, wsgi, celery)
│   ├── settings.py             # all configuration, read from the environment
│   ├── urls.py                 # root urlconf -> /api/
│   ├── celery.py               # Celery application
│   ├── wsgi.py / asgi.py
├── downloader/                 # the application
│   ├── views.py                # HTTP layer (thin: validate -> service -> JSON)
│   ├── urls.py                 # /api/analyze/, /api/download/, /api/files/…
│   ├── serializers.py          # request validation
│   ├── services.py             # business logic (analyse, download, merge, convert)
│   ├── extractor.py            # yt-dlp integration (options, errors, formats)
│   ├── policy.py               # SSRF protection for every yt-dlp request/redirect
│   ├── validators.py           # URL validation, host allow-list, DNS checks
│   ├── filenames.py            # safe file names
│   ├── ffmpeg.py               # safe FFmpeg wrapper (no shell, timeouts)
│   ├── models.py               # DownloadJob
│   ├── cache.py                # analysis / download / DNS caching
│   ├── tasks.py                # Celery tasks
│   ├── throttling.py           # rate limits
│   ├── cleanup.py              # expired files / stale temp files
│   ├── logging_utils.py        # log redaction (no signed URLs, no paths)
│   ├── exceptions.py           # error hierarchy + JSON envelope
│   ├── management/commands/cleanup_downloads.py
│   ├── migrations/
│   └── tests/                  # 184 tests
├── gunicorn.conf.py            # production WSGI configuration
├── Dockerfile                  # production image (with FFmpeg)
├── docker-compose.yml          # web + worker + beat + postgres + redis
├── docker-entrypoint.sh
├── requirements.txt
├── .env.example
└── README.md

The extraction logic never lives in a view – views.py only validates the request, calls a service and serialises the result.

Security

URL validation – only https (configurable), no credentials, only port 80/443, length limit, control characters removed, host allow-list per platform (youtube.com, youtu.be, tiktok.com, instagram.com, facebook.com, …). Look-alike domains such as youtube.com.evil.tld are rejected.
SSRF protection – the hostname is resolved via DNS and every returned IP must be a globally routable address. Loopback, private, link-local (incl. 169.254.169.254 cloud metadata), multicast and reserved ranges are blocked. The same check runs for every redirect that yt-dlp follows (downloader/policy.py patches the yt-dlp request handlers, so CDN hops are validated as well) and the number of redirects is capped.
Command injection – FFmpeg is always called with an argument list (subprocess.run([...]), never shell=True); file names are sanitised and paths are re-validated to stay inside the download directory.
Limits – socket timeout, extraction timeout, download timeout, max file size (checked before and during the download), max media duration.
Rate limiting – per client IP: 60 analyses/h, 20 downloads/h, 240 file requests/h (configurable) plus a global limit of parallel downloads.
Access control – every download link contains a random 256-bit token, is tied to a single job and expires. The file itself is never served from a public directory.
Clean-up – expired files, expired job rows and abandoned temporary files are removed automatically (see below).
Logging – a filter removes signed media URLs (sig=, token=, …), credentials in URLs and absolute filesystem paths from every log line.
No secrets in the API – the response never contains the secret key, the cookie file, the absolute path of a file or the source URL of a job.
CSRF – Django’s CSRF middleware is active, but the DRF views are csrf_exempt because the API is token-less and does not use cookies. Add the frontend origin to DJANGO_CSRF_TRUSTED_ORIGINS as soon as session based endpoints are added.

Known limitation: the DNS check resolves the hostname and then the HTTP client resolves it again. A very aggressive DNS rebinding attack is therefore theoretically possible. For high-risk deployments restrict the egress traffic of the server with a firewall (allow only the platform/CDN addresses) – this is the recommended production hardening.

Tests

python manage.py test                                   # everything (184 tests)
python manage.py test downloader.tests.test_validators  # single module
python manage.py test downloader.tests.test_cache       # the caching layer
CLIPFETCH_RUN_INTEGRATION_TESTS=1 python manage.py test downloader.tests.test_integration

The suite covers

valid / invalid / unsupported URLs and look-alike domains,
metadata extraction success and failure (extraction is mocked),
format selection and invalid format ids,
video downloads (progressive and merged), audio downloads (mp3/m4a/wav),
FFmpeg errors and FFmpeg-not-installed,
timeouts, file size limits and duration limits,
SSRF attempts (initial URL and redirects),
rate limiting and the download concurrency limit,
clean-up of expired downloads, old job rows and stale temporary files,
the caching layer (analysis, download and DNS cache, cache hits and misses, cache disabled, shared files that must survive the clean-up).

The integration tests use the real FFmpeg binary and (opt-in) the real network with a public, freely licensed Blender Foundation video.

Background jobs (Celery)

Downloads run inline by default – simple and good enough for a first release. For a production setup with many users, run them as background jobs:

# .env
CLIPFETCH_ASYNC=True
REDIS_URL=redis://localhost:6379/0
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0

celery -A config worker -l info     # worker
celery -A config beat -l info        # periodic clean-up

POST /api/download/ then answers with 202 Accepted and a status_url that the frontend polls. If the broker is unreachable the API automatically falls back to the synchronous mode.

Clean-up

Expired files are removed by three mechanisms:

Opportunistically – during API requests (at most every CLIPFETCH_CLEANUP_INTERVAL seconds, protected by a cache lock).
Management command – python manage.py cleanup_downloads (use it in a cron job or a systemd timer).
Celery beat – the downloader.cleanup_downloads_task runs every CLIPFETCH_CLEANUP_INTERVAL seconds in async mode.

The clean-up removes

files and job rows whose download link has expired,
job rows (and files) older than CLIPFETCH_JOB_RETENTION_DAYS,
temporary directories that were left behind by a crash (CLIPFETCH_TMP_MAX_AGE).

Intermediate files of a running download are always removed, even when the download fails.

Production deployment

0. Docker (recommended)

The repository contains a Dockerfile and a docker-compose.yml with the complete stack (web + Celery worker + Celery beat + PostgreSQL + Redis):

cp .env.example .env
# set DJANGO_SECRET_KEY, DJANGO_DEBUG=False, the passwords and CORS_ALLOWED_ORIGINS
docker compose up -d --build
docker compose logs -f web

The image installs FFmpeg, runs the migrations on start and exposes a /api/health/ health check. The media volume keeps the downloads.

Without Docker:

1. Environment 1. Environment

DJANGO_DEBUG=False
DJANGO_SECRET_KEY=<random 50 char string>
DJANGO_ALLOWED_HOSTS=api.example.com
CORS_ALLOWED_ORIGINS=https://app.example.com
POSTGRES_DB=clipfetch
POSTGRES_USER=clipfetch
POSTGRES_PASSWORD=<secret>
POSTGRES_HOST=127.0.0.1
REDIS_URL=redis://127.0.0.1:6379/0
CLIPFETCH_ASYNC=True
DJANGO_SECURE_SSL_REDIRECT=True
DJANGO_SECURE_HSTS_SECONDS=31536000
DJANGO_TRUST_PROXY_SSL_HEADER=True
DJANGO_NUM_PROXIES=1

2. Gunicorn

The repository contains a tuned gunicorn.conf.py (worker count, timeouts, recycling, access log format):

gunicorn config.wsgi:application -c gunicorn.conf.py

The timeout has to be larger than CLIPFETCH_DOWNLOAD_TIMEOUT when the downloads run synchronously. With CLIPFETCH_ASYNC=True the workers only queue the job and the default timeout is fine.

3. nginx (reverse proxy)

server {
    listen 80;
    server_name api.example.com;

    client_max_body_size 2m;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 900s;
    }
}

Set DJANGO_NUM_PROXIES=1 so that the rate limiter sees the real client IP.

4. systemd units

# /etc/systemd/system/clipfetch.service
[Unit]
Description=ClipFetch backend
After=network.target

[Service]
User=clipfetch
WorkingDirectory=/srv/clipfetch
EnvironmentFile=/srv/clipfetch/.env
ExecStart=/srv/clipfetch/.venv/bin/gunicorn config.wsgi:application \
          --bind 127.0.0.1:8000 --workers 4 --timeout 900
Restart=always

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/clipfetch-cleanup.service + .timer
[Service]
Type=oneshot
WorkingDirectory=/srv/clipfetch
EnvironmentFile=/srv/clipfetch/.env
ExecStart=/srv/clipfetch/.venv/bin/python manage.py cleanup_downloads

5. Data directories

mkdir -p /srv/clipfetch/media/downloads /srv/clipfetch/media/tmp
chown -R clipfetch:clipfetch /srv/clipfetch/media

6. Deploy

python manage.py migrate --noinput
python manage.py collectstatic --noinput
sudo systemctl restart clipfetch

7. Verified production numbers

Measured with the production stack (gunicorn, 4 workers × 4 threads, DEBUG=False, Redis cache, PostgreSQL):

Scenario	Result
Cached `POST /api/analyze/`	0.005 – 0.01 s
Cold `POST /api/analyze/` (TikTok / Instagram / Facebook)	0.5 – 1.9 s
Throughput (10 / 20 / 40 parallel requests)	220 / 231 / 234 req/s
p50 / p95 latency (40 parallel)	35 ms / 58 ms
Static overhead (`/api/health/`)	4 ms per request (249 req/s)
`python manage.py check --deploy`	no issues

Checklist

DJANGO_DEBUG=False and a unique DJANGO_SECRET_KEY
CORS_ALLOWED_ORIGINS contains exactly the frontend origin (never *)
DJANGO_SECURE_SSL_REDIRECT=True, DJANGO_SECURE_HSTS_SECONDS=31536000 and DJANGO_TRUST_PROXY_SSL_HEADER=True (TLS is terminated by the proxy)
PostgreSQL + Redis configured, CLIPFETCH_ASYNC=True with a running worker
REDIS_URL points to the Redis instance – required with more than one worker, otherwise the cache, the rate limits and the download concurrency limit are per process (the app logs a warning)
TLS terminated at the proxy
the clean-up timer/cron is active
FFmpeg installed on the server (/api/health/ → features.ffmpeg == true)
a firewall restricts the egress traffic of the server

Troubleshooting

Symptom	Cause	Fix
`ffmpeg_unavailable`	FFmpeg is not installed	`sudo apt install ffmpeg` (or use the Docker image), then check `GET /api/health/` → `features.ffmpeg`
`media_unavailable` on Instagram/Facebook	The content is private, region locked or the platform requires a login	Provide your own cookies: `CLIPFETCH_YTDLP_COOKIES_FILE=/path/cookies.txt`
`blocked_url`	The URL resolves to a private/internal address	Expected – the SSRF protection blocks it
`unsupported_url`	The platform is not on the allow-list	Only YouTube, TikTok, Instagram and Facebook are supported
`file_too_large`	The format exceeds `CLIPFETCH_MAX_DOWNLOAD_SIZE`	Pick a lower quality (the `too_large` flag in `/api/analyze/` marks them) or raise the limit
`429 rate_limited`	Too many requests from this IP	Wait for `Retry-After` or raise the `CLIPFETCH_RATE_*` values
`429 too_many_downloads`	The server is at its download concurrency limit	Raise `CLIPFETCH_MAX_CONCURRENT_DOWNLOADS`
`timeout`	The platform is slow or the file is huge	Lower the quality, raise `CLIPFETCH_DOWNLOAD_TIMEOUT` and `GUNICORN_TIMEOUT`
The download link returns `404`	The token is wrong or the file was already cleaned up	Request a new download
The download link returns `410`	The link expired (`CLIPFETCH_LINK_TTL`)	Request a new download
`extraction_failed`	yt-dlp could not extract the media	Check the server logs (the reason is logged), try another URL, try `CLIPFETCH_YTDLP_IMPERSONATE=chrome` or a JS runtime
The cache is not shared between the workers	`REDIS_URL` is not set	Set `REDIS_URL` – otherwise the cache, the rate limits and the download limit are per process
`400 Bad Request` in the browser console (CORS)	The frontend origin is not allowed	Add the exact origin to `CORS_ALLOWED_ORIGINS` and restart
`500` on every request in production	`DJANGO_SECRET_KEY` is missing	The app refuses to start with the development key – set a real one
The media list is empty	The platform changed its page/API	The fast path falls back to yt-dlp automatically; if that fails too, check the logs

Useful commands

python manage.py check --deploy        # production configuration check
python manage.py test                  # the test suite
python manage.py cleanup_downloads     # remove expired downloads now
curl -s localhost:8000/api/health/     # is FFmpeg available?
redis-cli keys "clipfetch:*"           # what is in the cache

Legal notice

This backend downloads media that the requesting user is allowed to access. Respect the terms of service of the platforms, the copyright of the content and the applicable law. The project does not bypass DRM, paywalls, age restrictions or any other access control, and it is not intended for redistributing copyrighted material.