← Back to blog

How to Scrape Dribbble Design Data with Python (2026)

2026-04-09 [python dribbble scraping design ui-ux cloudflare proxies sqlite trends]

How to Scrape Dribbble Design Data with Python (2026)

Dribbble is where designers share their work in progress — UI concepts, icons, illustrations, branding, and animations. The "shots" format (small previews of design work) makes it a concentrated source of visual design data. If you're building a design inspiration tool, studying UI trends, analyzing the job market for designers, or training a visual AI model, Dribbble has the data.

The platform offers an OAuth-based API with limited free access, and the web interface can be scraped for the rest. Here's how to work with both comprehensively, including SQLite storage, proxy integration, and building trend analysis pipelines.

API Setup

Dribbble's API requires registering an application at dribbble.com/account/applications. You'll get a client ID and secret. For read-only access to public data, generate a personal access token directly from the application settings page.

import httpx
import time
import re
import json
import sqlite3
import random
from datetime import datetime, timedelta
from collections import Counter

ACCESS_TOKEN = "your_dribbble_access_token"
BASE_URL = "https://api.dribbble.com/v2"

API_HEADERS = {
    "Authorization": f"Bearer {ACCESS_TOKEN}",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
}

WEB_HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
}


def dribbble_api(endpoint: str, params: dict = None) -> dict | list:
    """Make an authenticated request to the Dribbble API."""
    for attempt in range(3):
        try:
            response = httpx.get(
                f"{BASE_URL}/{endpoint}",
                headers=API_HEADERS,
                params=params or {},
                timeout=30,
            )

            if response.status_code == 429:
                # Check for Retry-After header
                retry_after = int(response.headers.get("Retry-After", 60))
                print(f"Rate limited. Waiting {retry_after}s...")
                time.sleep(retry_after)
                continue

            response.raise_for_status()
            return response.json()
        except httpx.HTTPStatusError as e:
            if attempt == 2:
                raise
            time.sleep(5 * (attempt + 1))

    return {}


def make_web_client(proxy_url: str = None) -> httpx.Client:
    """Create an httpx client for web scraping."""
    return httpx.Client(
        headers=WEB_HEADERS,
        proxy=proxy_url,
        timeout=30,
        follow_redirects=True,
        http2=True,
    )

Fetching Shots via API

The Dribbble API provides access to authenticated users' own shots and followed users' shots:

def get_user_shots(per_page: int = 30, page: int = 1) -> list[dict]:
    """Get the authenticated user's shots (API)."""
    data = dribbble_api("user/shots", {
        "per_page": per_page,
        "page": page,
        "sort": "recent",
    })

    return [
        {
            "id": s["id"],
            "title": s["title"],
            "description": s.get("description", ""),
            "html_url": s["html_url"],
            "views_count": s.get("views_count", 0),
            "likes_count": s.get("likes_count", 0),
            "comments_count": s.get("comments_count", 0),
            "buckets_count": s.get("buckets_count", 0),  # collections count
            "tags": s.get("tags", []),
            "image_hidpi": s.get("images", {}).get("hidpi"),
            "image_normal": s.get("images", {}).get("normal"),
            "image_teaser": s.get("images", {}).get("teaser"),
            "image_two_x": s.get("images", {}).get("two_x"),
            "published_at": s.get("published_at"),
            "updated_at": s.get("updated_at"),
            "animated": s.get("animated", False),
            "video": s.get("video") is not None,
            "attachment_id": s.get("attachments", [{}])[0].get("id") if s.get("attachments") else None,
        }
        for s in data if isinstance(data, list)
    ]


def get_all_user_shots() -> list[dict]:
    """Paginate through all of the authenticated user's shots."""
    all_shots = []
    page = 1

    while True:
        batch = get_user_shots(per_page=30, page=page)
        if not batch:
            break
        all_shots.extend(batch)
        print(f"  Page {page}: {len(batch)} shots (total: {len(all_shots)})")
        page += 1
        time.sleep(1)  # API rate limit: 60 requests/minute

    return all_shots


def get_shot_detail_api(shot_id: int) -> dict:
    """Get full details for a specific shot via API."""
    return dribbble_api(f"shots/{shot_id}")

Scraping the Public Feed

Dribbble's API is limited to authenticated user data. For the public feed, we work with the web interface:

def scrape_popular_shots(timeframe: str = "week", page: int = 1,
                          client: httpx.Client = None) -> list[dict]:
    """Scrape popular shots from Dribbble.

    timeframe: 'now', 'week', 'month', 'year', 'ever'
    """
    use_client = client or make_web_client()
    url = f"https://dribbble.com/shots/popular/{timeframe}"
    params = {"page": page}

    response = use_client.get(url, params=params)

    if response.status_code != 200:
        print(f"  Status {response.status_code} for page {page}")
        return []

    shots = []
    html = response.text

    # Dribbble embeds shot data as JSON in script tags
    json_pattern = re.compile(
        r'data-thumbnail-id="(\d+)"[^>]*>.*?'
        r'class="[^"]*shot-thumbnail-link[^"]*"[^>]*href="([^"]+)"',
        re.DOTALL,
    )

    # Try extracting from structured JSON-LD or inline data
    ld_pattern = re.compile(r'<script type="application/ld\+json">(.*?)</script>', re.DOTALL)
    for match in ld_pattern.finditer(html):
        try:
            data = json.loads(match.group(1))
            if isinstance(data, list):
                for item in data:
                    if item.get("@type") in ("ImageObject", "CreativeWork"):
                        shots.append({
                            "id": item.get("identifier", ""),
                            "title": item.get("name", ""),
                            "image_url": item.get("contentUrl") or item.get("thumbnailUrl", ""),
                            "url": item.get("url", ""),
                            "author": item.get("author", {}).get("name", ""),
                            "author_url": item.get("author", {}).get("url", ""),
                        })
        except json.JSONDecodeError:
            continue

    # Fallback: parse HTML shot cards
    if not shots:
        from bs4 import BeautifulSoup
        soup = BeautifulSoup(html, "lxml")

        for card in soup.select("li[class*='shot-thumbnail']"):
            shot_id = card.get("data-thumbnail-id", "")
            link = card.select_one("a[class*='thumbnail-link'], a[href*='/shots/']")
            img = card.select_one("img")
            likes_el = card.select_one("[class*='shot-likes-count'], [class*='likes']")
            views_el = card.select_one("[class*='shot-views-count'], [class*='views']")

            if link and shot_id:
                shots.append({
                    "id": shot_id,
                    "title": img.get("alt", "") if img else link.get_text(strip=True),
                    "url": link.get("href", ""),
                    "image_url": img.get("src", "") if img else "",
                    "likes": parse_count((likes_el.get_text(strip=True) if likes_el else "0")),
                    "views": parse_count((views_el.get_text(strip=True) if views_el else "0")),
                })

    return shots


def parse_count(text: str) -> int:
    """Parse display counts like '1.2k' or '2.3m' into integers."""
    text = text.strip().lower().replace(",", "").replace(" ", "")
    if not text or text == "-":
        return 0
    if text.endswith("k"):
        try:
            return int(float(text[:-1]) * 1_000)
        except ValueError:
            return 0
    if text.endswith("m"):
        try:
            return int(float(text[:-1]) * 1_000_000)
        except ValueError:
            return 0
    try:
        return int(text)
    except ValueError:
        return 0

Designer Profile Scraping

def scrape_designer_profile(username: str,
                              client: httpx.Client = None) -> dict:
    """Scrape a designer's public profile from Dribbble."""
    use_client = client or make_web_client()
    url = f"https://dribbble.com/{username}"
    response = use_client.get(url)

    if response.status_code == 404:
        return {"error": "Profile not found", "username": username}
    if response.status_code != 200:
        return {"error": f"HTTP {response.status_code}", "username": username}

    html = response.text

    def extract_meta(name: str, prop: str = "name") -> str:
        match = re.search(rf'<meta\s+{prop}="{re.escape(name)}"\s+content="([^"]*)"', html)
        return match.group(1) if match else ""

    # Extract stats — Dribbble shows shots, followers, following, likes
    stats = {}
    stat_pattern = re.compile(
        r'class="[^"]*count[^"]*"[^>]*>\s*([\d.,k]+)\s*</\w+>\s*'
        r'<[^>]+class="[^"]*label[^"]*"[^>]*>([^<]+)</\w+>',
        re.IGNORECASE,
    )
    for match in stat_pattern.finditer(html):
        stat_name = match.group(2).strip().lower()
        stats[stat_name] = parse_count(match.group(1))

    # Location
    location_match = re.search(r'"addressLocality"\s*:\s*"([^"]+)"', html)
    location = location_match.group(1) if location_match else ""

    # Availability for hire
    hire_match = re.search(r'"hiringStatus"\s*:\s*"([^"]+)"', html)
    hire_status = hire_match.group(1) if hire_match else ""

    # Skills/specialties
    skills = re.findall(r'href="/[^"]+"\s+class="[^"]*skill[^"]*">([^<]+)</a>', html)

    return {
        "username": username,
        "display_name": extract_meta("og:title", "property").replace(" on Dribbble", "").strip(),
        "bio": extract_meta("og:description", "property"),
        "location": location,
        "hire_status": hire_status,
        "url": url,
        "stats": stats,
        "skills": skills[:20],
        "avatar": extract_meta("og:image", "property"),
    }


def scrape_search_designers(query: str, page: int = 1,
                              client: httpx.Client = None) -> list[dict]:
    """Search for designers on Dribbble."""
    use_client = client or make_web_client()
    url = "https://dribbble.com/search/designers"
    params = {"q": query, "page": page}

    response = use_client.get(url, params=params)
    designers = []

    if response.status_code != 200:
        return designers

    html = response.text
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, "lxml")

    for card in soup.select("[class*='designer-card'], [class*='profile-card']"):
        username_link = card.select_one("a[href*='/'][class*='profile'], a[class*='username']")
        name_el = card.select_one("[class*='display-name'], h2, h3")
        followers_el = card.select_one("[class*='followers']")

        if username_link:
            href = username_link.get("href", "").strip("/")
            designers.append({
                "username": href.split("/")[-1],
                "display_name": name_el.get_text(strip=True) if name_el else "",
                "followers": parse_count(followers_el.get_text(strip=True) if followers_el else "0"),
                "url": f"https://dribbble.com/{href}",
            })

    return designers

Collecting Shot Details and Tags

def scrape_shot_detail(shot_id: str, client: httpx.Client = None) -> dict:
    """Scrape full details for a specific shot."""
    use_client = client or make_web_client()
    url = f"https://dribbble.com/shots/{shot_id}"
    response = use_client.get(url)

    if response.status_code != 200:
        return {"error": f"HTTP {response.status_code}", "id": shot_id}

    html = response.text

    # Extract structured data
    ld_match = re.search(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
    ld_data = {}
    if ld_match:
        try:
            ld_data = json.loads(ld_match.group(1))
        except json.JSONDecodeError:
            pass

    # Extract tags
    tags = re.findall(r'href="/tags/([a-z0-9\-]+)"', html)
    tags = list(dict.fromkeys(tags))  # Deduplicate preserving order

    # Extract color palette (Dribbble shows hex colors used in the design)
    colors = re.findall(r'(?:background-color|color)\s*:\s*#([0-9a-fA-F]{6})', html)
    # Also look for data attributes with hex colors
    hex_colors = re.findall(r'"(?:hex|color)"\s*:\s*"#?([0-9a-fA-F]{6})"', html)
    all_colors = list(dict.fromkeys(colors + hex_colors))[:8]

    # Stats from page
    views_match = re.search(r'"viewsCount"\s*:\s*(\d+)', html)
    likes_match = re.search(r'"likesCount"\s*:\s*(\d+)', html)
    comments_match = re.search(r'"commentsCount"\s*:\s*(\d+)', html)

    # Tools used (Figma, Sketch, etc.)
    tools = re.findall(r'(?:figma|sketch|adobe xd|photoshop|illustrator|framer)', html.lower())
    tools = list(set(tools))

    return {
        "id": shot_id,
        "title": ld_data.get("name", ""),
        "author": ld_data.get("author", {}).get("name", ""),
        "author_url": ld_data.get("author", {}).get("url", ""),
        "date_published": ld_data.get("datePublished", ""),
        "date_modified": ld_data.get("dateModified", ""),
        "description": ld_data.get("description", ""),
        "image_url": ld_data.get("image", ""),
        "url": url,
        "tags": tags,
        "colors": ["#" + c for c in all_colors],
        "tools": tools,
        "views": int(views_match.group(1)) if views_match else 0,
        "likes": int(likes_match.group(1)) if likes_match else 0,
        "comments": int(comments_match.group(1)) if comments_match else 0,
    }


def batch_scrape_shots(shot_ids: list[str], proxy_url: str = None,
                        delay_range: tuple = (2.0, 5.0)) -> list[dict]:
    """Scrape details for a list of shot IDs with rate limiting."""
    client = make_web_client(proxy_url)
    results = []

    for i, shot_id in enumerate(shot_ids):
        print(f"  [{i+1}/{len(shot_ids)}] Shot {shot_id}")
        detail = scrape_shot_detail(shot_id, client)

        if detail.get("title"):
            results.append(detail)
            print(f"    OK: {detail['title'][:50]} | Tags: {', '.join(detail['tags'][:4])}")
        else:
            print(f"    FAILED: {detail.get('error', 'unknown')}")

        if i < len(shot_ids) - 1:
            time.sleep(random.uniform(*delay_range))

    return results

Anti-Bot Measures

Dribbble has tightened its defenses significantly:

Aggressive rate limiting — The API allows 60 requests per minute. The web interface is even stricter — more than 20-30 page loads per minute from a single IP triggers a CAPTCHA wall or soft block.

Cloudflare with JS challenge — Dribbble uses Cloudflare's managed challenge mode. First-time visitors from suspicious IPs get a JavaScript challenge that headless browsers like basic Playwright setups can handle, but raw HTTP clients cannot pass without the right fingerprints.

Login walls — After viewing a handful of shots, Dribbble prompts visitors to log in. Unauthenticated scraping hits this wall. The content is still in the HTML (for SEO), but navigation between pages gets interrupted.

Fingerprint-based blocking — Dribbble tracks request patterns — sequential ID enumeration, uniform timing, missing Accept-Language headers — all trigger blocks. They've specifically hardened against ID crawling.

Search restrictions — Search results beyond page 10 require authentication. Tag pages have similar depth limits for anonymous users.

For any serious Dribbble collection, rotating residential IPs are essential. ThorData's proxy infrastructure passes Cloudflare's residential IP checks and rotates automatically, which prevents pattern-based blocking:

THORDATA_PROXY = "http://USER:[email protected]:9000"

def scrape_with_rotation(shot_ids: list[str]) -> list[dict]:
    """Scrape shots through rotating residential proxies."""
    # Create fresh client every 25 requests to get a new proxy rotation
    results = []
    batch_size = 25

    for batch_start in range(0, len(shot_ids), batch_size):
        batch = shot_ids[batch_start:batch_start + batch_size]
        client = make_web_client(THORDATA_PROXY)

        for shot_id in batch:
            response = client.get(f"https://dribbble.com/shots/{shot_id}")
            if response.status_code == 200:
                detail = scrape_shot_detail(shot_id, client)
                if detail.get("title"):
                    results.append(detail)
                    print(f"OK: {shot_id} — {detail['title'][:40]}")
            elif response.status_code in [403, 429]:
                print(f"Blocked: {shot_id} — status {response.status_code}")
                time.sleep(30)  # Back off on block
            else:
                print(f"Error: {shot_id} — status {response.status_code}")

            time.sleep(random.uniform(2.0, 5.0))

    return results

SQLite Storage

def init_dribbble_db(db_path: str = "dribbble_data.db") -> sqlite3.Connection:
    """Initialize SQLite database for Dribbble data."""
    conn = sqlite3.connect(db_path)
    conn.execute("PRAGMA journal_mode=WAL")

    conn.execute("""
        CREATE TABLE IF NOT EXISTS shots (
            id TEXT PRIMARY KEY,
            title TEXT,
            author TEXT,
            author_url TEXT,
            date_published TEXT,
            description TEXT,
            image_url TEXT,
            url TEXT,
            views INTEGER DEFAULT 0,
            likes INTEGER DEFAULT 0,
            comments INTEGER DEFAULT 0,
            tags TEXT,       -- JSON array
            colors TEXT,     -- JSON array of hex colors
            tools TEXT,      -- JSON array of design tools mentioned
            animated INTEGER DEFAULT 0,
            timeframe TEXT,  -- 'week', 'month', etc. if from popular feed
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)

    conn.execute("""
        CREATE TABLE IF NOT EXISTS designers (
            username TEXT PRIMARY KEY,
            display_name TEXT,
            bio TEXT,
            location TEXT,
            hire_status TEXT,
            avatar TEXT,
            shots_count INTEGER,
            followers_count INTEGER,
            following_count INTEGER,
            likes_count INTEGER,
            skills TEXT,  -- JSON array
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)

    conn.execute("""
        CREATE TABLE IF NOT EXISTS tag_trends (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            tag TEXT NOT NULL,
            shot_count INTEGER,
            avg_likes REAL,
            avg_views REAL,
            recorded_week TEXT,  -- ISO week format: YYYY-Www
            recorded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)

    conn.execute("""
        CREATE TABLE IF NOT EXISTS color_trends (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            hex_color TEXT NOT NULL,
            occurrence_count INTEGER,
            avg_likes REAL,
            recorded_week TEXT,
            recorded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)

    conn.execute("CREATE INDEX IF NOT EXISTS idx_shots_author ON shots(author)")
    conn.execute("CREATE INDEX IF NOT EXISTS idx_shots_published ON shots(date_published)")
    conn.execute("CREATE INDEX IF NOT EXISTS idx_shots_likes ON shots(likes)")
    conn.execute("CREATE INDEX IF NOT EXISTS idx_tag_trends_week ON tag_trends(recorded_week)")

    conn.commit()
    return conn


def save_shot(conn: sqlite3.Connection, shot: dict) -> bool:
    """Save a shot to the database. Returns True if new."""
    existing = conn.execute("SELECT 1 FROM shots WHERE id = ?", (shot["id"],)).fetchone()

    conn.execute(
        """INSERT OR REPLACE INTO shots
           (id, title, author, author_url, date_published, description, image_url,
            url, views, likes, comments, tags, colors, tools, animated, timeframe)
           VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
        (
            shot["id"], shot.get("title"), shot.get("author"), shot.get("author_url"),
            shot.get("date_published"), shot.get("description"), shot.get("image_url"),
            shot.get("url"), shot.get("views", 0), shot.get("likes", 0),
            shot.get("comments", 0), json.dumps(shot.get("tags", [])),
            json.dumps(shot.get("colors", [])), json.dumps(shot.get("tools", [])),
            1 if shot.get("animated") else 0, shot.get("timeframe"),
        )
    )
    conn.commit()
    return not existing


def record_tag_trends(conn: sqlite3.Connection, shots: list[dict]) -> None:
    """Record weekly tag trends from a batch of shots."""
    from collections import defaultdict

    week = datetime.now().strftime("%Y-W%V")
    tag_stats = defaultdict(lambda: {"count": 0, "total_likes": 0, "total_views": 0})

    for shot in shots:
        for tag in shot.get("tags", []):
            tag_stats[tag]["count"] += 1
            tag_stats[tag]["total_likes"] += shot.get("likes", 0)
            tag_stats[tag]["total_views"] += shot.get("views", 0)

    for tag, stats in tag_stats.items():
        conn.execute(
            """INSERT INTO tag_trends (tag, shot_count, avg_likes, avg_views, recorded_week)
               VALUES (?, ?, ?, ?, ?)""",
            (
                tag,
                stats["count"],
                stats["total_likes"] / stats["count"] if stats["count"] > 0 else 0,
                stats["total_views"] / stats["count"] if stats["count"] > 0 else 0,
                week,
            )
        )
    conn.commit()

Trend Analysis

def top_tags_by_engagement(db_path: str, days: int = 30,
                             min_shots: int = 5) -> list:
    """Get tags with highest average engagement in recent shots."""
    conn = sqlite3.connect(db_path)
    # We need to unnest the JSON tags array - use Python since SQLite doesn't have unnest
    rows = conn.execute(
        """SELECT tags, likes, views FROM shots
           WHERE scraped_at >= datetime('now', ?)
             AND tags != '[]'""",
        (f"-{days} days",)
    ).fetchall()
    conn.close()

    tag_stats = {}
    for (tags_json, likes, views) in rows:
        try:
            tags = json.loads(tags_json or "[]")
            for tag in tags:
                if tag not in tag_stats:
                    tag_stats[tag] = {"count": 0, "total_likes": 0, "total_views": 0}
                tag_stats[tag]["count"] += 1
                tag_stats[tag]["total_likes"] += (likes or 0)
                tag_stats[tag]["total_views"] += (views or 0)
        except json.JSONDecodeError:
            continue

    results = [
        {
            "tag": tag,
            "shot_count": stats["count"],
            "avg_likes": stats["total_likes"] / stats["count"],
            "avg_views": stats["total_views"] / stats["count"],
        }
        for tag, stats in tag_stats.items()
        if stats["count"] >= min_shots
    ]

    return sorted(results, key=lambda x: x["avg_likes"], reverse=True)[:30]


def trending_colors(db_path: str, days: int = 14) -> list:
    """Find most frequently used colors in recent high-engagement shots."""
    conn = sqlite3.connect(db_path)
    rows = conn.execute(
        """SELECT colors, likes FROM shots
           WHERE scraped_at >= datetime('now', ?)
             AND colors != '[]'
             AND likes > 50""",
        (f"-{days} days",)
    ).fetchall()
    conn.close()

    color_counts = Counter()
    for (colors_json, likes) in rows:
        try:
            colors = json.loads(colors_json or "[]")
            # Weight by likes
            weight = max(1, likes // 50)
            for color in colors:
                color_counts[color.upper()] += weight
        except json.JSONDecodeError:
            continue

    return [
        {"color": color, "weighted_count": count}
        for color, count in color_counts.most_common(20)
    ]


def designer_leaderboard(db_path: str, min_shots: int = 10) -> list:
    """Rank designers by average shot engagement."""
    conn = sqlite3.connect(db_path)
    rows = conn.execute(
        """SELECT author, COUNT(*) as shot_count,
                  AVG(likes) as avg_likes, MAX(likes) as max_likes,
                  AVG(views) as avg_views, SUM(likes) as total_likes
           FROM shots
           WHERE author != ''
           GROUP BY author
           HAVING COUNT(*) >= ?
           ORDER BY avg_likes DESC
           LIMIT 30""",
        (min_shots,)
    ).fetchall()
    conn.close()

    return [
        {
            "author": row[0],
            "shot_count": row[1],
            "avg_likes": round(row[2], 1),
            "max_likes": row[3],
            "avg_views": round(row[4], 1),
            "total_likes": row[5],
        }
        for row in rows
    ]

Full Collection Pipeline

def run_weekly_trend_collection(
    db_path: str = "dribbble_data.db",
    proxy_url: str = None,
    timeframes: list[str] = None,
) -> dict:
    """Weekly pipeline: scrape popular shots and update trend data."""
    if timeframes is None:
        timeframes = ["week", "month"]

    conn = init_dribbble_db(db_path)
    client = make_web_client(proxy_url)
    stats = {"total_shots": 0, "new_shots": 0, "failed": 0}

    for timeframe in timeframes:
        print(f"\n=== Popular: {timeframe} ===")

        for page in range(1, 6):  # 5 pages per timeframe
            shots_list = scrape_popular_shots(timeframe, page, client)
            if not shots_list:
                print(f"  No shots on page {page}, stopping")
                break

            print(f"  Page {page}: {len(shots_list)} shots")
            shot_ids = [s["id"] for s in shots_list if s.get("id")]

            # Get details for each shot
            for shot_summary in shots_list:
                shot_id = shot_summary.get("id", "")
                if not shot_id:
                    continue

                # Check if we already have recent data
                existing = conn.execute(
                    """SELECT 1 FROM shots WHERE id = ?
                       AND scraped_at >= datetime('now', '-3 days')""",
                    (str(shot_id),)
                ).fetchone()

                if existing:
                    stats["total_shots"] += 1
                    continue

                detail = scrape_shot_detail(str(shot_id), client)
                detail["timeframe"] = timeframe

                if detail.get("title"):
                    is_new = save_shot(conn, detail)
                    stats["total_shots"] += 1
                    if is_new:
                        stats["new_shots"] += 1
                else:
                    stats["failed"] += 1

                time.sleep(random.uniform(2, 4))

            time.sleep(random.uniform(5, 10))

        # Record tag and color trends from this timeframe
        all_shots = [
            dict(zip(
                ["id", "tags", "colors", "likes", "views"],
                row
            ))
            for row in conn.execute(
                """SELECT id, tags, colors, likes, views FROM shots
                   WHERE timeframe = ? AND scraped_at >= datetime('now', '-1 day')""",
                (timeframe,)
            ).fetchall()
        ]
        record_tag_trends(conn, all_shots)

    conn.close()

    print(f"\n=== Summary ===")
    print(f"  Total shots processed: {stats['total_shots']}")
    print(f"  New shots saved: {stats['new_shots']}")
    print(f"  Failed: {stats['failed']}")

    return stats


if __name__ == "__main__":
    PROXY = "http://USER:[email protected]:9000"
    run_weekly_trend_collection(proxy_url=PROXY)

Practical Tips

  1. Focus on tags. Dribbble's tag system reveals what's trending in design. Track tag frequency week over week to spot emerging patterns like "glassmorphism", "bento grid", or "AI-generated".
  2. Color analysis. The inline color palette data is free metadata. Build seasonal or trend-based color reports — useful for marketing teams and brand designers.
  3. Animated shots filter. Filter for GIF/video shots (animated: true in API) to track motion design trends separately from static work.
  4. Rate your delays. The login wall and CAPTCHA trigger on velocity, not volume. Slow, irregular requests (2-5 second jitter) collect more data than fast bursts that get blocked after 50 requests.
  5. The JSON-LD is your friend. When it's present, it contains cleaner structured data than HTML parsing and is less likely to break on redesigns.
  6. Dribbble's tag pages (dribbble.com/tags/{tag}) are the best source for category-specific trends. They paginate up to ~10 pages for unauthenticated users.
  7. Profile stats decay fast. Designer follower counts and shot likes change quickly. If you need accurate point-in-time data, timestamp your snapshots and don't treat old scraped data as current.

Dribbble is a concentrated source of professional design data. With respectful rate limiting and proxy rotation from ThorData, you can build valuable datasets for design trend analysis, visual AI training, and creative market research.