Bibliometric Analysis Workflow: Deduplicating Scopus, Web of Science, and OpenAlex CSVs

Combining multi-database bibliographic exports requires rigorous deduplication, entity normalization, and reproducible network modeling beyond trivial citation counts.

HumanPen Team

· 11 min read

Direct answer: building a reproducible multi-database bibliometric workflow

To conduct a rigorous bibliometric analysis across Web of Science, Scopus, and OpenAlex, you must normalize and deduplicate records by lowercase DOI and fuzzy title matching, standardize author names and affiliations, and separate performance metrics from science mapping. Simply plotting citation counts or uncleaned keyword frequencies produces distorted networks; valid field mapping requires harmonizing citation schemas, applying fractional or full counting transparently, and computing reproducible co-citation or thematic evolution models.

Methodological standard: Bibliometric reviews evaluated in peer review must document exact database export dates, search strings, deduplication loss counts, and disambiguation rules. Presenting raw citation metrics without cross-database deduplication or treating author name variants as independent scholars invalidates network centrality and cluster boundaries.

In contemporary scholarly synthesis, relying on a single bibliographic index restricts research coverage and introduces commercial indexing bias. However, merging exports across disparate bibliographic repositories inevitably collides with mismatched metadata schemas, missing identifiers, and conflicting citation counts. Before synthesizing field trends, isolating data preparation for an auditable review ensures empirical reliability, mirroring the systematic evidence checks established in our systematic review standards for AI detection literature. Furthermore, because downstream network models depend directly on the integrity of underlying cited handles, researchers should verify external metadata using our guide on verifying academic citations, DOIs, and URLs.

Comparing metadata schemas across Web of Science, Scopus, and OpenAlex

Each scholarly indexing service formats bibliographic records under distinct structural schemas, export boundaries, and citation-linkage paradigms. Web of Science provides structured plain text and BibTeX exports with traditional two-character field tags such as `AU` for authors, `TI` for titles, and `CR` for cited references, but enforces a strict per-batch export limit of 500 records for full records with cited references. Scopus offers comma-separated CSV and BibTeX files with richer multidisciplinary conference coverage and native author identifiers, yet limits web downloads to 2,000 items and structures multi-author strings with internal semicolons. Meanwhile, OpenAlex offers a completely open, API-native bibliographic graph covering over 250 million works, linking papers via persistent identifiers like ROR institution IDs and Wikidata concepts rather than static publisher text strings.

Database PlatformExport Formats & Field SchemaUnique Analytical AdvantageIdentifier & Disambiguation StackIngestion & Harmonization Challenges
Web of Science (Core Collection)Plain text, BibTeX (`AU`, `TI`, `SO`, `CR`, `UT`)Curated core journal quality; rigorous historical cited reference indexingResearcherID, ORCID; curated source lists500-record batch export ceiling; proprietary two-letter field tags require specialized parsers
Scopus (Elsevier)CSV, BibTeX (`Authors`, `Title`, `Source`, `Cited by`, `References`)Broad multidisciplinary scope; strong non-English and conference coverageScopus Author ID, Affiliation ID; algorithmic profile disambiguationSemicolon-delimited multi-value cells; 2,000-record web export boundary
OpenAlex (OurResearch)JSON-LD, CSV via API (`id`, `doi`, `authorships`, `referenced_works`)Completely open catalog; programmatic API endpoints; unrestricted bulk retrievalOpenAlex ID, ROR, ORCID, Crossref DOI, WikidataCitation edges point to internal OpenAlex IDs rather than raw reference text strings

Harmonizing these three sources requires building an intermediate schema that resolves structural discrepancies without losing source-specific intelligence. For example, while Web of Science lists cited references as semi-structured text strings inside the `CR` field (such as `SMITH J, 2020, J BUS RES, V110, P1`), Scopus provides formatted titles and publication venues in its `References` column, and OpenAlex records cited works as an array of unique entity URLs. If an analyst naively merges raw CSVs into a single spreadsheet without transforming cited reference strings into a normalized structure, network algorithms will fail to calculate co-citation frequencies across the combined dataset.

Multi-pass deduplication, author disambiguation, and counting methods

Because prominent journal articles frequently index across both Scopus and Web of Science simultaneously, multi-database bibliometrics demands a robust, auditable deduplication pipeline. Deduplication must proceed across multiple passes: an exact primary match on normalized Digital Object Identifiers, followed by a secondary fuzzy-string match on titles and publication years for non-indexed proceedings or historical books. Relying solely on DOIs drops unindexed monographs, while relying solely on string matching produces false positives when papers share standard editorial titles like "Introduction to the Special Issue" or "Errata".

  1. DOI standardization and canonicalization: Strip URL prefixes (such as `https://doi.org/` or `http://dx.doi.org/`), convert all characters to lowercase, and trim trailing whitespace, quotation marks, or punctuation to create a uniform primary key.
  2. Multi-pass title and author matching for non-DOI entries: Group records lacking DOIs by exact publication year, normalize titles by stripping non-alphanumeric characters and stop words, and compute normalized Levenshtein edit distance ratios (accepting matches with similarity >= 0.95 and matching first-author surnames).
  3. Entity disambiguation with authority thesauri: Merge author name variants (such as `Aria, M.` and `Aria, Massimo`) using ORCID identifiers, OpenAlex author profiles, or custom thesaurus term harmonization tables, while standardizing institutional affiliations using Research Organization Registry (ROR) canonical IDs.
  4. Transparent counting method selection: Explicitly document whether publication and citation tallies employ full counting (awarding one whole credit to every contributing author and institution) or fractional counting (dividing credit equally as `1 / n` among `n` co-authors).
Counting bias caution: As documented by Donthu et al. (2021), full counting inflates aggregate publication volume when collaborative multi-institution research teams grow, biasing performance rankings toward large collaborative consortiums. Applying fractional counting preserves total systemic weight across the corpus, but requires clean co-authorship attribution tables to avoid fractional allocation errors.

Beyond author names, keyword normalization prevents severe network fragmentation in co-word analysis. Authors routinely use acronyms, hyphens, and plural inflections interchangeably (for example, `bibliometric analysis`, `bibliometrics`, `scientometrics`, and `science mapping`). Creating a two-column synonym thesaurus mapping raw variants to a single consolidated descriptor ensures that conceptual clusters reflect substantive domain themes rather than trivial lexical choices. This synonym normalization must be stored as an auditable text file, allowing peer reviewers to inspect every lexical grouping.

Performance analysis versus science mapping: metrics, co-citation, and thematic evolution

In their foundational methodological framework, Donthu et al. (2021) and Aria & Cuccurullo (2017) emphasize that rigorous bibliometric research consists of two complementary tracks: performance analysis and science mapping. Performance analysis examines the productivity and scientific impact of constituent actors (authors, institutions, countries, and journals) through quantitative metrics like publication growth rates, citation counts, and h-index distributions. In contrast, science mapping uncovers the structural, cognitive, and social architecture of a scientific field through network analysis, spatial layouts, and clustering algorithms.

Mapping DimensionUnderlying Network UnitCore Analytical ObjectiveTheoretical Interpretation FrameworkPrimary Computational Algorithm
Co-Citation AnalysisCited reference pairs in `CR` columnUncover the historical foundations and intellectual traditions of a domainHighly co-cited papers represent shared theoretical paradigms and foundational methodsAssociation strength normalization; Louvain community detection
Bibliographic CouplingShared references in current papersMap contemporary active research fronts and emerging paradigmsCurrent papers citing identical literature share active methodological and thematic frontiersJaccard similarity; Leiden algorithm with resolution optimization
Co-Word & Thematic EvolutionAuthor keywords and title phrasesTrace conceptual trajectory and structural shifts across time slicesCallon centrality (external influence) versus density (internal conceptual cohesion)Strategic coordinate diagrams; temporal alluvial/Sankey stream flows
Co-Authorship MappingAuthors, departments, and countriesEvaluate research collaboration patterns and international networksNetwork density, betweenness centrality, and structural holes identify key knowledge brokersForceAtlas2 spatial layout; degree centrality distribution

To analyze how scientific paradigms shift across decades, researchers divide cleaned corpora into consecutive time windows (such as 3-year or 5-year periods) and construct thematic evolution maps. As implemented in `bibliometrix` and Python network libraries, calculating Callon centrality (measuring the intensity of external links to other themes) and Callon density (measuring the internal cohesion among constituent keywords) categorizes topics into a four-quadrant strategic diagram. Motor themes (high centrality, high density) represent well-developed core drivers of the field; basic themes (high centrality, low density) denote transversal foundational topics; niche themes (low centrality, high density) reflect highly specialized isolated subfields; and emerging or declining themes (low centrality, low density) highlight frontier opportunities or fading topics.

Executing an end-to-end bibliometric pipeline in HumanPen Start Project

Conducting a defensible bibliometric review involves coordinating multiple software packages, resolving memory bottlenecks when parsing millions of citation edges, and ensuring that no records are silently truncated during spreadsheet imports. Rather than wrestling with manual script errors or unverified black-box desktop tools, researchers can structure their analysis through HumanPen Start Project. The platform accepts raw data archives in Excel, CSV, Word, PDF, or zip formats up to 2 GB total, running analytical code inside an isolated sandbox environment using Python or R. Databases requiring local institutional access, such as CNKI or specialized commercial registries, can be exported locally and uploaded directly for standardized processing.

Attach your data and papers, describe what you need, approve the plan before work starts, and get the reviewed report, charts, and code ready to preview online or download.
  1. Raw data ingest and audit plan confirmation: Upload exported CSV and BibTeX files from Web of Science, Scopus, and OpenAlex alongside your research questions, review the proposed multi-pass deduplication and disambiguation plan, and approve execution parameters before computational billing begins.
  2. Deterministic multi-database deduplication and audit logging: The sandbox executes automated Python scripts to normalize DOIs, compute fuzzy string matching across missing identifiers, generate a deduplicated master CSV dataset, and output an auditable log accounting for every retained and removed record.
  3. Performance metrics synthesis and Bradford zone modeling: Automated pipelines compute annual scientific production, author productivity distributions (Lotka's law), core source journals (Bradford's law zones 1 to 3), and h-index matrices, exported as publication-ready tables.
  4. Science mapping network modeling: Construct co-citation, bibliographic coupling, co-word thematic evolution, and international collaboration networks using deterministic modularity algorithms, exported as publication-grade vector graphics (PDF/SVG) and interactive HTML matrices.
  5. Multi-asset delivery and reproducible script archiving: Preview the comprehensive analytical report, summary tables, and high-resolution figures online, or download the full deliverable bundle containing clean datasets, `requirements.txt`, and executable `run_all.py` analysis scripts.

As demonstrated across our secondary statistical analysis example project and grant literature scoping template on `/projects`, all generated reports, network figures, and analytical tables serve strictly as an empirical reference scaffold and reproducible research draft. Authors retain complete scientific responsibility for their research synthesis: researchers must independently verify that search queries align with PRISMA guidelines, inspect the lexical validity of keyword thesauri, supply personal domain insights, and declare computational tool assistance transparently in accordance with journal and university editorial policies.

Frequently asked questions about multi-database bibliometrics

How should I resolve discrepancies when the same article reports conflicting citation counts in Scopus and Web of Science? Citation counts naturally diverge across platforms because Scopus, Web of Science, and OpenAlex index distinct journal portfolios and publication types. Rather than averaging disparate counts into an arbitrary synthetic number, researchers should record the retrieval date and either designate a single primary benchmark database for citation metrics (such as Scopus for broader coverage or Web of Science Core Collection for journal conservatism) or retain them as distinct variables (such as `scopus_citations` and `wos_citations`) within the deduplicated master CSV.

Why does neglecting author name disambiguation distort co-authorship and citation networks? Authors frequently publish under multiple name variants, such as `Smith, J.`, `Smith, John`, and `Smith, J. A.`, while distinct scholars frequently share identical surnames and initials. Failing to normalize these variants splits a prolific researcher into multiple artificial nodes with depressed centrality scores, while homonyms falsely merge distinct researchers into phantom collaborative hubs. Resolving this distortion requires mapping records against persistent identifiers like ORCID, Scopus Author ID, and OpenAlex author IDs, complemented by manual verification for dominant authors in the field.

What is the fundamental difference between co-citation analysis and bibliographic coupling? Co-citation analysis measures the frequency with which two past documents are cited together by subsequent publications, uncovering the historical foundations and theoretical traditions of a discipline. In contrast, bibliographic coupling measures the extent to which two contemporary documents cite the same prior references, grouping papers currently exploring active research fronts. Importantly, while co-citation connections evolve dynamically over time as new literature cites past works, bibliographic coupling relationships remain fixed once papers are published.

Can Chinese-language literature from CNKI be combined with Web of Science exports for unified science mapping? Yes, but CNKI data requires format standardization before network integration. Standard CNKI exports via Refworks or EndNote lack Web of Science two-character field tags and omit structured citation links for international journals. To integrate them, analysts must parse CNKI records into a unified tabular schema (aligning title, author, abstract, and publication year), map bilingual author keywords using a cross-language synonym thesaurus, and clearly distinguish Chinese journal core coverage from international indices in the methodology section.

Sources and verified primary literature

The multi-database ingestion protocols, disambiguation algorithms, and science mapping frameworks detailed in this guide are grounded in established bibliometric literature and technical documentation:

KEEP READING