AIStoreHub Package Vault: Whisper
Package Slug: whisper
Archetype: tool
Upstream Repository: https://github.com/openai/whisper
Workspace Files & Specifications (25 files)
- aistorehub.json (410 bytes)
{ "name": "Whisper", "slug": "whisper", "version": "1.0.0", "type": "tool", "description": "Robust Speech Recognition via Large-Scale Weak Supervision", "owner": "community", "repository": "catalog", "github_repo_url": "https://github.com/openai/whisper", "tags": [ "trending", "openai", "python", "github", "open-source" ], "imported_at": "2026-11-21T23:56:31.909Z" } - CHANGELOG.md (9024 bytes)
# CHANGELOG ## [v20250625](https://github.com/openai/whisper/releases/tag/v20250625) * Fix: Update torch.load to use weights_only=True to prevent security w… ([#2451](https://github.com/openai/whisper/pull/2451)) * Fix: Ensure DTW cost tensor is on the same device as input tensor ([#2561](https://github.com/openai/whisper/pull/2561)) * docs: updated README to specify translation model limitation ([#2547](https://github.com/openai/whisper/pull/2547)) * Fixed triton kernel update to support latest triton versions ([#2588](https://github.com/openai/whisper/pull/2588)) * Fix: GitHub display errors for Jupyter notebooks ([#2589](https://github.com/openai/whisper/pull/2589)) * Bump the github-actions group with 3 updates ([#2592](https://github.com/openai/whisper/pull/2592)) * Keep GitHub Actions up to date with GitHub's Dependabot ([#2486](https://github.com/openai/whisper/pull/2486)) * pre-commit: Upgrade black v25.1.0 and isort v6.0.0 ([#2514](https://github.com/openai/whisper/pull/2514)) * GitHub Actions: Add Python 3.13 to the testing ([#2487](https://github.com/openai/whisper/pull/2487)) * PEP 621: Migrate from setup.py to pyproject.toml ([#2435](https://github.com/openai/whisper/pull/2435)) * pre-commit autoupdate && pre-commit run --all-files ([#2484](https://github.com/openai/whisper/pull/2484)) * Upgrade GitHub Actions ([#2430](https://github.com/openai/whisper/pull/2430)) * Bugfix: Illogical "Avoid computing higher temperatures on no_speech" ([#1903](https://github.com/openai/whisper/pull/1903)) * Updating README and doc strings to reflect that n_mels can now be 128 ([#2049](https://github.com/openai/whisper/pull/2049)) * fix typo data/README.md ([#2433](https://github.com/openai/whisper/pull/2433)) * Update README.md ([#2379](https://github.com/openai/whisper/pull/2379)) * Add option to carry initial_prompt with the sliding window ([#2343](https://github.com/openai/whisper/pull/2343)) * more pytorch versions in tests ([#2408](https://github.com/openai/whisper/pull/2408)) ## [v20240930](https://github.com/openai/whisper/releases/tag/v20240930) * allowing numpy 2 in tests ([#2362](https://github.com/openai/whisper/pull/2362)) * large-v3-turbo model ([#2361](https://github.com/openai/whisper/pull/2361)) * test on python/pytorch versions up to 3.12 and 2.4.1 ([#2360](https://github.com/openai/whisper/pull/2360)) * using sdpa if available ([#2359](https://github.com/openai/whisper/pull/2359)) ## [v20240927](https://github.com/openai/whisper/releases/tag/v20240927) * pinning numpy<2 in tests ([#2332](https://github.com/openai/whisper/pull/2332)) * Relax triton requirements for compatibility with pytorch 2.4 and newer ([#2307](https://github.com/openai/whisper/pull/2307)) * Skip silence around hallucinations ([#1838](https://github.com/openai/whisper/pull/1838)) * Fix triton env marker ([#1887](https://github.com/openai/whisper/pull/1887)) ## [v20231117](https://github.com/openai/whisper/releases/tag/v20231117) * Relax triton requirements for compatibility with pytorch 2.1 and newer ([#1802](https://github.com/openai/whisper/pull/1802)) ## [v20231106](https://github.com/openai/whisper/releases/tag/v20231106) * large-v3 ([#1761](https://github.com/openai/whisper/pull/1761)) ## [v20231105](https://github.com/openai/whisper/releases/tag/v20231105) * remove tiktoken pin ([#1759](https://github.com/openai/whisper/pull/1759)) * docs: Disambiguation of the term "relative speed" in the README ([#1751](https://github.com/openai/whisper/pull/1751)) * allow_pickle=False while loading of mel matrix IN audio.py ([#1511](https://github.com/openai/whisper/pull/1511)) * handling transcribe exceptions. ([#1682](https://github.com/openai/whisper/pull/1682)) * Add new option to generate subtitles by a specific number of words ([#1729](https://github.com/openai/whisper/pull/1729)) * Fix exception when an audio file with no speech is provided ([#1396](https://github.com/openai/whisper/pull/1396)) ## [v20230918](https://github.com/openai/whisper/releases/tag/v20230918) * Add .pre-commit-config.yaml ([#1528](https://github.com/openai/whisper/pull/1528)) * fix doc of TextDecoder ([#1526](https://github.com/openai/whisper/pull/1526)) * Update model-card.md ([#1643](https://github.com/openai/whisper/pull/1643)) * word timing tweaks ([#1559](https://github.com/openai/whisper/pull/1559)) * Avoid rearranging all caches ([#1483](https://github.com/openai/whisper/pull/1483)) * Improve timestamp heuristics. ([#1461](https://github.com/openai/whisper/pull/1461)) * fix condition_on_previous_text ([#1224](https://github.com/openai/whisper/pull/1224)) * Fix numba depreceation notice ([#1233](https://github.com/openai/whisper/pull/1233)) * Updated README.md to provide more insight on BLEU and specific appendices ([#1236](https://github.com/openai/whisper/pull/1236)) * Avoid computing higher temperatures on no_speech segments ([#1279](https://github.com/openai/whisper/pull/1279)) * Dropped unused execute bit from mel_filters.npz. ([#1254](https://github.com/openai/whisper/pull/1254)) * Drop ffmpeg-python dependency and call ffmpeg directly. ([#1242](https://github.com/openai/whisper/pull/1242)) * Python 3.11 ([#1171](https://github.com/openai/whisper/pull/1171)) * Update decoding.py ([#1219](https://github.com/openai/whisper/pull/1219)) * Update decoding.py ([#1155](https://github.com/openai/whisper/pull/1155)) * Update README.md to reference tiktoken ([#1105](https://github.com/openai/whisper/pull/1105)) * Implement max line width and max line count, and make word highlighting optional ([#1184](https://github.com/openai/whisper/pull/1184)) * Squash long words at window and sentence boundaries. ([#1114](https://github.com/openai/whisper/pull/1114)) * python-publish.yml: bump actions version to fix node warning ([#1211](https://github.com/openai/whisper/pull/1211)) * Update tokenizer.py ([#1163](https://github.com/openai/whisper/pull/1163)) ## [v20230314](https://github.com/openai/whisper/releases/tag/v20230314) * abort find_alignment on empty input ([#1090](https://github.com/openai/whisper/pull/1090)) * Fix truncated words list when the replacement character is decoded ([#1089](https://github.com/openai/whisper/pull/1089)) * fix github language stats getting dominated by jupyter notebook ([#1076](https://github.com/openai/whisper/pull/1076)) * Fix alignment between the segments and the list of words ([#1087](https://github.com/openai/whisper/pull/1087)) * Use tiktoken ([#1044](https://github.com/openai/whisper/pull/1044)) ## [v20230308](https://github.com/openai/whisper/releases/tag/v20230308) * kwargs in decode() for convenience ([#1061](https://github.com/openai/whisper/pull/1061)) * fix all_tokens handling that caused more repetitions and discrepancy in JSON ([#1060](https://github.com/openai/whisper/pull/1060)) * fix typo in CHANGELOG.md ## [v20230307](https://github.com/openai/whisper/releases/tag/v20230307) * Fix the repetition/hallucination issue identified in #1046 ([#1052](https://github.com/openai/whisper/pull/1052)) * Use triton==2.0.0 ([#1053](https://github.com/openai/whisper/pull/1053)) * Install triton in x86_64 linux only ([#1051](https://github.com/openai/whisper/pull/1051)) * update setup.py to specify python >= 3.8 requirement ## [v20230306](https://github.com/openai/whisper/releases/tag/v20230306) * remove auxiliary audio extension ([#1021](https://github.com/openai/whisper/pull/1021)) * apply formatting with `black`, `isort`, and `flake8` ([#1038](https://github.com/openai/whisper/pull/1038)) * word-level timestamps in `transcribe()` ([#869](https://github.com/openai/whisper/pull/869)) * Decoding improvements ([#1033](https://github.com/openai/whisper/pull/1033)) * Update README.md ([#894](https://github.com/openai/whisper/pull/894)) * Fix infinite loop caused by incorrect timestamp tokens prediction ([#914](https://github.com/openai/whisper/pull/914)) * drop python 3.7 support ([#889](https://github.com/openai/whisper/pull/889)) ## [v20230124](https://github.com/openai/whisper/releases/tag/v20230124) * handle printing even if sys.stdout.buffer is not available ([#887](https://github.com/openai/whisper/pull/887)) * Add TSV formatted output in transcript, using integer start/end time in milliseconds ([#228](https://github.com/openai/whisper/pull/228)) * Added `--output_format` option ([#333](https://github.com/openai/whisper/pull/333)) * Handle `XDG_CACHE_HOME` properly for `download_root` ([#864](https://github.com/openai/whisper/pull/864)) * use stdout for printing transcription progress ([#867](https://github.com/openai/whisper/pull/867)) * Fix bug where mm is mistakenly replaced with hmm in e.g. 20mm ([#659](https://github.com/openai/whisper/pull/659)) * print '?' if a letter can't be encoded using the system default encoding ([#859](https://github.com/openai/whisper/pull/859)) ## [v20230117](https://github.com/openai/whisper/releases/tag/v20230117) The first versioned release available on [PyPI](https://pypi.org/project/openai-whisper/)
- data/meanwhile.json (67802 bytes)
{ "1YOmY-Vjy-o": { "begin": "1:04.0", "end": "2:11.0", "text": "FOLKS, IF YOU WATCH THE SHOW,\nYOU KNOW I SPEND A LOT OF TIME\nRIGHT OVER THERE, PATIENTLY AND\nASTUTELY SCRUTINIZING THE\nBOXWOOD AND MAHOGANY CHESS SET\nOF THE DAY'S BIGGEST STORIES,\nDEVELOPING THE CENTRAL\nHEADLINE-PAWNS, DEFTLY\nMANEUVERING AN OH-SO-TOPICAL\nKNIGHT TO F-6, FEIGNING A\nCLASSIC SICILIAN-NAJDORF\nVARIATION ON THE NEWS, ALL THE\nWHILE, SEEING EIGHT MOVES DEEP\nAND PATIENTLY MARSHALING THE\nLATEST PRESS RELEASES INTO A\nFISCHER SOZIN LIPNITZKY ATTACK\nTHAT CULMINATES IN THE ELEGANT,\nLETHAL, SLOW-PLAYED EN PASSANT\nCHECKMATE THAT IS MY NIGHTLY\nMONOLOGUE.\nBUT SOMETIMES, SOMETIMES,\nFOLKS-- I,\nSOMETIMES,\nI STARTLE AWAKE UPSIDE DOWN ON\nTHE MONKEY BARS OF A CONDEMNED\nPLAYGROUND ON A SUPERFUND SITE,\nGET ALL HEPPED UP ON GOOFBALLS,\nRUMMAGE THROUGH A DISCARDED TAG\nBAG OF DEFECTIVE TOYS, YANK\nOUT A FISTFUL OF DISEMBODIED\nDOLL LIMBS, TOSS THEM ON A\nSTAINED KID'S PLACEMAT FROM A\nDEFUNCT DENNY'S, SET UP A TABLE\nINSIDE A RUSTY CARGO CONTAINER\nDOWN BY THE WHARF, AND CHALLENGE\nTOOTHLESS DRIFTERS TO THE\nGODLESS BUGHOUSE BLITZ\nOF TOURNAMENT OF NEWS THAT IS MY\nSEGMENT:\nMEANWHILE!\n" }, "3P_XnxdlXu0": { "begin": "2:08.3", "end": "3:02.3", "text": "FOLKS, I SPEND A LOT OF TIME\nRIGHT OVER THERE, NIGHT AFTER NIGHT ACTUALLY, CAREFULLY\nSELECTING FOR YOU THE DAY'S NEWSIEST,\nMOST AERODYNAMIC HEADLINES,\nSTRESS TESTING THE MOST TOPICAL\nANTILOCK BRAKES AND POWER\nSTEERING, PAINSTAKINGLY\nSTITCHING LEATHER SEATING SO\nSOFT, IT WOULD MAKE J.D. POWER\nAND HER ASSOCIATES BLUSH, TO\nCREATE THE LUXURY SEDAN THAT IS\nMY NIGHTLY MONOLOGUE.\nBUT SOMETIMES, JUST SOMETIMES,\nFOLKS, I LURCH TO CONSCIOUSNESS\nIN THE BACK OF AN ABANDONED\nSCHOOL BUS AND SLAP MYSELF AWAKE\nWITH A CRUSTY FLOOR MAT BEFORE\nUSING A MOUSE-BITTEN TIMING BELT\nTO STRAP SOME OLD PLYWOOD TO A\nCOUPLE OF DISCARDED OIL DRUMS.\nTHEN, BY THE LIGHT OF A HEATHEN\nMOON, RENDER A GAS TANK OUT OF\nAN EMPTY BIG GULP, FILL IT WITH\nWHITE CLAW AND DENATURED\nALCOHOL, THEN LIGHT A MATCH AND\nLET HER RIP, IN THE DEMENTED\nONE-MAN SOAP BOX DERBY OF NEWS\nTHAT IS MY SEGMENT: MEANWHILE!" }, "3elIlQzJEQ0": { "begin": "1:08.5", "end": "1:58.5", "text": "LADIES AND GENTLEMEN, YOU KNOW, I SPEND A\nLOT OF TIME RIGHT OVER THERE,\nRAISING THE FINEST HOLSTEIN NEWS\nCATTLE, FIRMLY, YET TENDERLY,\nMILKING THE LATEST HEADLINES\nFROM THEIR JOKE-SWOLLEN TEATS,\nCHURNING THE DAILY STORIES INTO\nTHE DECADENT, PROVENCAL-STYLE\nTRIPLE CREME BRIE THAT IS MY\nNIGHTLY MONOLOGUE.\nBUT SOMETIMES, SOMETIMES, FOLKS,\nI STAGGER HOME HUNGRY AFTER\nBEING RELEASED BY THE POLICE,\nAND ROOT AROUND IN THE NEIGHBOR'S\nTRASH CAN FOR AN OLD MILK\nCARTON, SCRAPE OUT THE BLOOMING\nDAIRY RESIDUE ONTO THE REMAINS\nOF A WET CHEESE RIND I WON\nFROM A RAT IN A PRE-DAWN STREET\nFIGHT, PUT IT IN A DISCARDED\nPAINT CAN, AND LEAVE IT TO\nFERMENT NEXT TO A TRASH FIRE,\nTHEN HUNKER DOWN AND HALLUCINATE\nWHILE EATING THE LISTERIA-LADEN\nDEMON CUSTARD OF NEWS THAT IS\nMY SEGMENT: MEANWHILE!" }, "43P4q1KGKEU": { "begin": "0:29.3", "end": "1:58.3", "text": "FOLKS, IF YOU WATCH THE SHOW, YOU KNOW I SPEND MOST\nOF MY TIME, RIGHT OVER THERE.\nCAREFULLY SORTING THROUGH THE\nDAY'S BIGGEST STORIES, AND\nSELECTING ONLY THE MOST\nSUPPLE AND UNBLEMISHED OSTRICH\nAND CROCODILE NEWS LEATHER,\nWHICH I THEN ENTRUST TO ARTISAN\nGRADUATES OF THE \"ECOLE\nGREGOIRE-FERRANDI,\" WHO\nCAREFULLY DYE THEM IN A PALETTE\nOF BRIGHT, ZESTY SHADES, AND\nADORN THEM WITH THE FINEST, MOST\nTOPICAL INLAY WORK USING HAND\nTOOLS AND DOUBLE MAGNIFYING\nGLASSES, THEN ASSEMBLE THEM\nACCORDING TO NOW CLASSIC AND\nELEGANT GEOMETRY USING OUR\nSIGNATURE SADDLE STITCHING, AND\nLINE IT WITH BEESWAX-COATED\nLINEN, AND FINALLY ATTACH A\nMALLET-HAMMERED STRAP, PEARLED\nHARDWARE, AND A CLOCHETTE TO\nCREATE FOR YOU THE ONE-OF-A-KIND\nHAUTE COUTURE HERMES BIRKIN BAG\nTHAT IS MY MONOLOGUE.\nBUT SOMETIMES, SOMETIMES, FOLKS,\nSOMETIMES, SOMETIMES I WAKE UP IN THE\nLAST CAR OF AN ABANDONED ROLLER\nCOASTER AT CONEY ISLAND, WHERE\nI'M HIDING FROM THE TRIADS, I\nHUFF SOME ENGINE LUBRICANTS OUT\nOF A SAFEWAY BAG AND STAGGER\nDOWN THE SHORE TO TEAR THE SAIL\nOFF A BEACHED SCHOONER, THEN I\nRIP THE CO-AXIAL CABLE OUT OF\nTHE R.V. OF AN ELDERLY COUPLE\nFROM UTAH, HANK AND MABEL,\nLOVELY FOLKS, AND USE IT TO\nSTITCH THE SAIL INTO A LOOSE,\nPOUCH-LIKE RUCKSACK, THEN I\nSTOW AWAY IN THE BACK OF A\nGARBAGE TRUCK TO THE JUNK YARD\nWHERE I PICK THROUGH THE DEBRIS\nFOR ONLY THE BROKEN TOYS THAT\nMAKE ME THE SADDEST UNTIL I HAVE\nLOADED, FOR YOU, THE HOBO\nFUGITIVE'S BUG-OUT BINDLE OF\nNEWS THAT IS MY SEGMENT:\nMEANWHILE!" }, "4ktyaJkLMfo": { "begin": "0:42.5", "end": "1:26.5", "text": "YOU KNOW, FOLKS, I SPEND A LOT\nOF TIME CRAFTING FOR YOU A\nBESPOKE PLAYLIST OF THE DAY'S\nBIGGEST STORIES, RIGHT OVER THERE, METICULOUSLY\nSELECTING THE MOST TOPICAL\nCHAKRA-AFFIRMING SCENTED\nCANDLES, AND USING FENG SHUI TO\nPERFECTLY ALIGN THE JOKE ENERGY\nIN THE EXCLUSIVE BOUTIQUE YOGA\nRETREAT THAT IS MY MONOLOGUE.\nBUT SOMETIMES, JUST SOMETIMES,\nI GO TO THE DUMPSTER BEHIND THE\nWAFFLE HOUSE AT 3:00 IN THE\nMORNING, TAKE OFF MY SHIRT,\nCOVER MYSELF IN USED FRY OIL,\nWRAP MY HANDS IN SOME OLD DUCT\nTAPE I STOLE FROM A BROKEN CAR\nWINDOW, THEN POUND A SIX-PACK OF\nBLUEBERRY HARD SELTZER AND A\nSACK OF PILLS I STOLE FROM A\nPARKED AMBULANCE, THEN\nARM-WRESTLE A RACCOON IN THE\nBACK ALLEY VISION QUEST OF NEWS\nTHAT IS MY SEGMENT:\n\"MEANWHILE!\"" }, "5Dsh9AgqRG0": { "begin": "1:06.0", "end": "2:34.0", "text": "YOU KNOW, FOLKS, I SPEND MOST OF\nMY TIME RIGHT OVER THERE, MINING\nTHE DAY'S BIGGEST, MOST\nIMPORTANT STORIES, COLLECTING\nTHE FINEST, MOST TOPICAL IRON\nORE, HAND HAMMERING IT INTO JOKE\nPANELS.\nTHEN I CRAFT SHEETS OF BRONZE\nEMBLAZONED WITH PATTERNS THAT\nTELL AN EPIC TALE OF CONQUEST\nAND GLORY.\nTHEN, USING THE GERMANIC\nTRADITIONAL PRESSBLECH\nPROCESS, I PLACE THIN SHEETS OF\nFOIL AGAINST THE SCENES, AND BY\nHAMMERING OR OTHERWISE,\nAPPLYING PRESSURE FROM THE BACK,\nI PROJECT THESE SCENES INTO A\nPAIR OF CHEEK GUARDS AND A\nFACEPLATE.\nAND, FINALLY, USING FLUTED\nSTRIPS OF WHITE ALLOYED\nMOULDING, I DIVIDE THE DESIGNS\nINTO FRAMED PANELS AND HOLD IT\nALL TOGETHER USING BRONZE RIVETS\nTO CREATE THE BEAUTIFUL AND\nINTIMIDATING ANGLO-SAXON\nBATTLE HELM THAT IS MY\nNIGHTLY MONOLOGUE.\nBUT SOMETIMES, SOMETIMES, FOLKS,\nSOMETIMES, JUST SOMETIMES, I COME TO MY SENSES FULLY NAKED\nON THE DECK OF A PIRATE-BESIEGED\nMELEE CONTAINER SHIP THAT PICKED\nME UP FLOATING ON THE DETACHED\nDOOR OF A PORT-A-POTTY IN THE\nINDIAN OCEAN.\nTHEN, AFTER A SUNSTROKE-INDUCED\nREALIZATION THAT THE CREW OF\nTHIS SHIP PLANS TO SELL ME IN\nEXCHANGE FOR A BAG OF ORANGES TO\nFIGHT OFF SCURVY, I LEAD A\nMUTINY USING ONLY A P.V.C. PIPE\nAND A POOL CHAIN.\nTHEN, ACCEPTING MY NEW ROLE AS\nCAPTAIN, AND DECLARING MYSELF\nKING OF THE WINE-DARK SEAS, I\nGRAB A DIRTY MOP BUCKET COVERED\nIN BARNACLES AND ADORN IT WITH\nTHE TEETH OF THE VANQUISHED, TO\nCREATE THE SOPPING WET PIRATE\nCROWN OF NEWS THAT IS MY\nSEGMENT:\n\"MEANWHILE!\" " }, "748OyesQy84": { "begin": "0:40.0", "end": "1:41.0", "text": "FOLKS, IF YOU WATCH THE SHOW, YOU KNOW, I SPEND MOST OF\nMY TIME, RIGHT OVER THERE,\nCAREFULLY BLENDING FOR YOU THE\nDAY'S NEWSIEST, MOST TOPICAL\nFLOUR, EGGS, MILK, AND BUTTER,\nAND STRAINING IT INTO A FINE\nBATTER TO MAKE DELICATE, YET\nINFORMATIVE COMEDY PANCAKES.\nTHEN I GLAZE THEM IN THE JUICE\nAND ZEST OF THE MOST RELEVANT\nMIDNIGHT VALENCIA ORANGES, AND\nDOUSE IT ALL IN A FINE DELAMAIN\nDE VOYAGE COGNAC, BEFORE\nFLAMBEING AND BASTING THEM TABLE\nSIDE, TO SERVE FOR YOU THE JAMES\nBEARD AWARD-WORTHY CREPES\nSUZETTE THAT IS MY NIGHTLY,\nMONOLOGUE.\nBUT SOMETIMES, JUST SOMETIMES FOLKS, I WAKE UP IN THE\nBAGGAGE HOLD OF A GREYHOUND BUS\nAS ITS BEING HOISTED BY THE\nSCRAPYARD CLAW TOWARD THE BURN\nPIT, ESCAPE TO A NEARBY\nABANDONED PRICE CHOPPER, WHERE I\nSCROUNGE FOR OLD BREAD SCRAPS,\nBUSTED OPEN BAGS OF STAR FRUIT\nCANDIES, AND EXPIRED EGGS,\nCHUCK IT ALL IN A DIRTY HUBCAP\nAND SLAP IT OVER A TIRE FIRE\nBEFORE USING THE LEGS OF A\nSTAINED PAIR OF SWEATPANTS AS\nOVEN MITTS TO EXTRACT AND SERVE\nTHE DEMENTED TRANSIENT'S POUND\nCAKE OF NEWS THAT IS MY SEGMENT:\n\"MEANWHILE.\"" }, "8prs9Pq5Xhk": { "begin": "1:18.5", "end": "2:17.5", "text": "FOLKS, IF YOU WATCH THE SHOW,\nAND I HOPE YOU DO, I SPEND A\nLOT OF TIME RIGHT OVER THERE,\nTIRELESSLY STUDYING THE LINEAGE\nOF THE DAY'S MOST IMPORTANT\nTHOROUGHBRED STORIES AND\nHOLSTEINER HEADLINES, WORKING\nWITH THE BEST TRAINERS MONEY CAN\nBUY TO REAR THEIR COMEDY\nOFFSPRING WITH A HAND THAT IS\nSTERN, YET GENTLE, INTO THE\nTRIPLE-CROWN-WINNING EQUINE\nSPECIMEN THAT IS MY NIGHTLY\nMONOLOGUE.\nBUT SOMETIMES, SOMETIMES, FOLKS\nI BREAK INTO AN UNINCORPORATED\nVETERINARY GENETICS LAB AND GRAB\nWHATEVER TEST TUBES I CAN FIND.\nAND THEN, UNDER A GROW LIGHT I GOT\nFROM A DISCARDED CHIA PET, I MIX\nTHE PILFERED D.N.A. OF A HORSE\nAND WHATEVER WAS IN A TUBE\nLABELED \"KEITH-COLON-EXTRA,\"\nSLURRYING THE CONCOCTION WITH\nCAFFEINE PILLS AND A MICROWAVED\nRED BULL, I SCREAM-SING A PRAYER\nTO JANUS, INITIATOR OF HUMAN\nLIFE AND GOD OF TRANSFORMATION\nAS A HALF-HORSE, HALF-MAN FREAK,\nSEIZES TO LIFE BEFORE ME IN THE\nHIDEOUS COLLECTION OF LOOSE\nANIMAL PARTS AND CORRUPTED MAN\nTISSUE THAT IS MY SEGMENT:\nMEANWHILE!" }, "9gX4kdFajqE": { "begin": "0:44.0", "end": "1:08.0", "text": "FOLKS, IF YOU WATCH THE SHOW,\nYOU KNOW SOMETIMES I'M OVER\nTHERE DOING THE MONOLOGUE.\nAND THEN THERE'S A COMMERCIAL\nBREAK, AND THEN I'M SITTING\nHERE.\nAND I DO A REALLY LONG\nDESCRIPTION OF A DIFFERENT\nSEGMENT ON THE SHOW, A SEGMENT\nWE CALL... \"MEANWHILE!\"" }, "9ssGpE9oem8": { "begin": "0:00.0", "end": "0:58.0", "text": "WELCOME BACK, EVERYBODY.\nYOU KNOW, FOLKS, I SPEND MOST OF\nMY TIME RIGHT OVER THERE,\nCOMBING OVER THE DAY'S NEWS,\nSELECTING ONLY THE HIGHEST\nQUALITY AND MOST TOPICAL BONDED\nCALFSKIN-LEATHER STORIES,\nCAREFULLY TANNING THEM AND CUTTING\nTHEM WITH MILLIMETER PRECISION,\nTHEN WEAVING IT TOGETHER IN\nA DOUBLE-FACED INTRECCIATO\nPATTERN TO CREATE FOR YOU THE\nEXQUISITE BOTTEGA VENETA CLUTCH\nTHAT IS MY MONOLOGUE.\nBUT SOMETIMES, WHILE AT A RAVE\nIN A CONDEMNED CEMENT FACTORY, I\nGET INJECTED WITH A MYSTERY\nCOCKTAIL OF HALLUCINOGENS AND\nPAINT SOLVENTS, THEN, OBEYING\nTHE VOICES WHO WILL STEAL MY\nTEETH IF I DON'T, I STUMBLE INTO\nA SHIPYARD WHERE I RIP THE\nCANVAS TARP FROM A GRAVEL TRUCK,\nAND TIE IT OFF WITH THE ROPE FROM A\nROTTING FISHING NET, THEN WANDER\nA FOOD COURT, FILLING IT WITH\nWHAT I THINK ARE GOLD COINS BUT\nARE, IN FACT, OTHER PEOPLE'S CAR\nKEYS, TO DRAG AROUND THE\nROOTLESS TRANSIENT'S CLUSTER\nSACK OF NEWS THAT IS MY SEGMENT:\n\"MEANWHILE!\"" }, "ARw4K9BRCAE": { "begin": "0:26.0", "end": "1:16.0", "text": "YOU KNOW, FOLKS, I SPEND\nMOST OF MY TIME STANDING RIGHT OVER\nTHERE,\nGOING OVER THE DAY'S NEWS\nAND SELECTING THE FINEST,\nMOST\nTOPICAL CARBON FIBER\nSTORIES, SHAPING THEM IN DRY\nDOCK INTO A\nSLEEK AND SEXY HULL, KITTING\nIT OUT WITH THE MOST TOPICAL\nFIBERGLASS AND TEAK FITTINGS\nBRASS RAILINGS, HOT TUBS,\nAND\nNEWS HELIPADS, TO CREATE THE\nCUSTOM DESIGNED, GLEAMING\nMEGA-YACHT THAT IS MY NIGHTLY\nMONOLOGUE. BUT SOMETIMES, JUST SOMETIMES FOLKS, I\nWASH ASHORE AT\nAN ABANDONED BEACH RESORT\nAFTER A NIGHT OF BATH SALTS\nAND\nSCOPOLAMINE, LASH SOME\nROTTING PICNIC TABLES\nTOGETHER, THEN\nDREDGE THE NEWS POND TO\nHAUL UP WHATEVER DISCARDED\nTRICYCLES AND BROKEN\nFRISBEES I CAN FIND, STEAL\nAN EYE PATCH\nFROM A HOBO, STAPLE A DEAD\nPIGEON TO MY SHOULDER, AND\nSAIL\nINTO INTERNATIONAL WATERS\nON THE PIRATE GARBAGE SCOW\nOF\nNEWS THAT IS MY SEGMENT:\nMEANWHILE!" }, "B1DRmrOlKtY": { "begin": "1:34.0", "end": "2:17.0", "text": "FOLKS, I SPEND A LOT OF TIME\nSTANDING RIGHT OVER THERE, OKAY,\nHANDPICKING ONLY THE RIPEST\nMOST TOPICAL DAILY HEADLINES,\nSEEKING OUT THE SWEETEST, MOST\nREFINED OF JEST-JAMS AND\nJOKE-JELLIES, CURATING A PLATE\nOF LOCAL COMEDY-FED MEATS AND\nSATIRICAL CHEESES TO LAY OUT THE\nARTISANAL CHARCUTERIE BOARD\nA-NEWS BOUCHE THAT IS MY NIGHTLY\nMONOLOGUE.\nBUT SOMETIMES, I WAKE UP IN THE\nGREASE TRAP OF AN\nUNLICENSED SLAUGHTERHOUSE,\nSPLASH MY FACE WITH SOME BEEF TALLOW,\nRENDER A RUDIMENTARY PRESERVE\nFROM BONE MARROW AND MELTED\nGUMMY WORMS, AND THROW IT\nTOGETHER WITH SOME SWEETBREADS\nAND TRIPE TO PRESENT THE\nPLOWMAN'S PLATTER OF UNCLAIMED\nOFFAL THAT IS MY SEGMENT:\nMEANWHILE!" }, "BT950jqCCUY": { "begin": "1:00.0", "end": "2:09.0", "text": "YOU KNOW, FOLKS,\nI SPEND SO MUCH OF MY TIME RIGHT\nOVER THERE, SIFTING THROUGH THE\nDAY'S BIGGEST STORIES, HAND\nSELECTING ONLY THE FINEST, MOST\nPERFECTLY AGED BURMESE NEWS\nTEAK.\nTHEN CAREFULLY CARVING AND\nSHAPING IT INTO A REFINED\nBUDDHA, DEPICTED IN THE \"CALLING\nTHE EARTH TO WITNESS\" GESTURE,\nWITH CHARACTERISTIC CIRCULAR\nPATTERNS ON THE ROBE, SHINS, AND\nKNEES, OF COURSE, WHICH I\nTHEN CAREFULLY GILD WITH THE\nMOST TOPICAL GOLD LEAF.\nTHEN I HARVEST THE SAP OF A\nUSITATA TREE TO APPLY THE DELICATE\nTAI O LACQUER, AND FINALLY HAND\nDECORATE IT WITH THE WHITE GLASS\nINLAYS TO CREATE FOR YOU THE\nGLORIOUS AMARA-PURA PERIOD\nSCULPTURE THAT IS MY MONOLOGUE.\nBUT SOMETIMES, SOMETIMES FOLKS,\nI WASH UP ON A GALVESTON\nBEACH, NAKED ON A RAFT OF GAS\nCANS AFTER ESCAPING FROM A FIGHT CLUB\nIN INTERNATIONAL WATERS.\nTHEN, STILL DERANGED ON A\nCOCKTAIL OF MESCALINE AND COUGH\nSYRUP, I STEAL A CINDER BLOCK\nFROM UNDER A STRIPPED '87 FORD\nTAURUS, AND CHIP AWAY AT IT\nWITH A BROKEN UMBRELLA HANDLE I\nSCAVENGED FROM A GOODWILL\nDUMPSTER UNTIL IT VAGUELY\nRESEMBLES THE HUNGERING WOLF\nTHAT SCRATCHES AT THE DOOR\nOF MY DREAMS AND PRESENT TO YOU\nTHE TORMENTED DREAD-EFFIGY OF\nNEWS THAT IS MY SEGMENT:\nMEANWHILE!" }, "C0e8XM30tQI": { "begin": "0:57.0", "end": "2:40.0", "text": "YOU KNOW, IF YOU WATCH THE SHOW, YOU'RE AWARE THAT I SPEND MOST OF\nMY TIME RIGHT OVER THERE,\nWANDERING THE NEWS FOREST FOR\nYOU, FELLING ONLY THE BIGGEST\nAND HEARTIEST WHITE STORY OAKS,\nCUTTING AND SHAPING THEM INTO\nTHE NEWSIEST, MOST TOPICAL\nCLEATS, CLAMPS AND PLANKS,\nKEEPING THEM AT A CONSTANT\nANGLE, GRADUALLY CREATING A\nSHELL-SHAPED, SHALLOW BOWL HULL\nUSING THE FIRE-BENDING TECHNIQUE\nINSTEAD OF STEAM-BENDING\nOBVIOUSLY.\nTHEN I LAY OUT ALL THE KEEL\nBLOCKS TO CAREFULLY SET UP THE\nSTEM, STERN AND, GARBOARD,\nATTACH THE BILGE FUTTOCKS TO THE\nTIMBER AND LOVINGLY CRAFT A FLAT\nTRANSOM STERN OUT OF\nNATURALLY-CURVED QUARTER\nCIRCLES.\nTHEN SECURE ALL THE PLANKS WITH\nTRUNNELS HANDMADE FROM THE\nFINEST LOCUST WOOD AND, FINALLY,\nADORN IT WITH A PROUD BOWSPRIT,\nFOREPEAK, AND CUSTOM GILDED\nFIGUREHEAD TO PRESENT TO YOU THE\nDUTCH GOLDEN - data/README.md (6211 bytes)
This directory supplements the paper with more details on how we prepared the data for evaluation, to help replicate our experiments. ## Short-form English-only datasets ### LibriSpeech We used the test-clean and test-other splits from the [LibriSpeech ASR corpus](https://www.openslr.org/12). ### TED-LIUM 3 We used the test split of [TED-LIUM Release 3](https://www.openslr.org/51/), using the segmented manual transcripts included in the release. ### Common Voice 5.1 We downloaded the English subset of Common Voice Corpus 5.1 from [the official website](https://commonvoice.mozilla.org/en/datasets) ### Artie We used the [Artie bias corpus](https://github.com/artie-inc/artie-bias-corpus). This is a subset of the Common Voice dataset. ### CallHome & Switchboard We used the two corpora from [LDC2002S09](https://catalog.ldc.upenn.edu/LDC2002S09) and [LDC2002T43](https://catalog.ldc.upenn.edu/LDC2002T43) and followed the [eval2000_data_prep.sh](https://github.com/kaldi-asr/kaldi/blob/master/egs/fisher_swbd/s5/local/eval2000_data_prep.sh) script for preprocessing. The `wav.scp` files can be converted to WAV files with the following bash commands: ```bash mkdir -p wav while read name cmd; do echo $name echo ${cmd/\|/} wav/$name.wav | bash done < wav.scp ``` ### WSJ We used [LDC93S6B](https://catalog.ldc.upenn.edu/LDC93S6B) and [LDC94S13B](https://catalog.ldc.upenn.edu/LDC94S13B) and followed the [s5 recipe](https://github.com/kaldi-asr/kaldi/tree/master/egs/wsj/s5) to preprocess the dataset. ### CORAAL We used the 231 interviews from [CORAAL (v. 2021.07)](https://oraal.uoregon.edu/coraal) and used the segmentations from [the FairSpeech project](https://github.com/stanford-policylab/asr-disparities/blob/master/input/CORAAL_transcripts.csv). ### CHiME-6 We downloaded the [CHiME-5 dataset](https://spandh.dcs.shef.ac.uk//chime_challenge/CHiME5/download.html) and followed the stage 0 of the [s5_track1 recipe](https://github.com/kaldi-asr/kaldi/tree/master/egs/chime6/s5_track1) to create the CHiME-6 dataset which fixes synchronization. We then used the binaural recordings (`*_P??.wav`) and the corresponding transcripts. ### AMI-IHM, AMI-SDM1 We preprocessed the [AMI Corpus](https://groups.inf.ed.ac.uk/ami/corpus/overview.shtml) by following the stage 0 and 2 of the [s5b recipe](https://github.com/kaldi-asr/kaldi/tree/master/egs/ami/s5b). ## Long-form English-only datasets ### TED-LIUM 3 To create a long-form transcription dataset from the [TED-LIUM3](https://www.openslr.org/51/) dataset, we sliced the audio between the beginning of the first labeled segment and the end of the last labeled segment of each talk, and we used the concatenated text as the label. Below are the timestamps used for slicing each of the 11 TED talks in the test split. | Filename | Begin time (s) | End time (s) | |---------------------|----------------|--------------| | DanBarber_2010 | 16.09 | 1116.24 | | JaneMcGonigal_2010 | 15.476 | 1187.61 | | BillGates_2010 | 15.861 | 1656.94 | | TomWujec_2010U | 16.26 | 402.17 | | GaryFlake_2010 | 16.06 | 367.14 | | EricMead_2009P | 18.434 | 536.44 | | MichaelSpecter_2010 | 16.11 | 979.312 | | DanielKahneman_2010 | 15.8 | 1199.44 | | AimeeMullins_2009P | 17.82 | 1296.59 | | JamesCameron_2010 | 16.75 | 1010.65 | | RobertGupta_2010U | 16.8 | 387.03 | ### Meanwhile This dataset consists of 64 segments from The Late Show with Stephen Colbert. The YouTube video ID, start and end timestamps, and the labels can be found in [meanwhile.json](meanwhile.json). The labels are collected from the closed-caption data for each video and corrected with manual inspection. ### Rev16 We use a subset of 16 files from the 30 podcast episodes in [Rev.AI's Podcast Transcription Benchmark](https://www.rev.ai/blog/podcast-transcription-benchmark-part-1/), after finding that there are multiple cases where a significant portion of the audio and the labels did not match, mostly on the parts introducing the sponsors. We selected 16 episodes that do not have this error, whose "file number" are: 3 4 9 10 11 14 17 18 20 21 23 24 26 27 29 32 ### Kincaid46 This dataset consists of 46 audio files and the corresponding transcripts compiled in the blog article [Which automatic transcription service is the most accurate - 2018](https://medium.com/descript/which-automatic-transcription-service-is-the-most-accurate-2018-2e859b23ed19) by Jason Kincaid. We used the 46 audio files and reference transcripts from the Airtable widget in the article. For the human transcription benchmark in the paper, we use a subset of 25 examples from this data, whose "Ref ID" are: 2 4 5 8 9 10 12 13 14 16 19 21 23 25 26 28 29 30 33 35 36 37 42 43 45 ### Earnings-21, Earnings-22 For these datasets, we used the files available in [the speech-datasets repository](https://github.com/revdotcom/speech-datasets), as of their `202206` version. ### CORAAL We used the 231 interviews from [CORAAL (v. 2021.07)](https://oraal.uoregon.edu/coraal) and used the full-length interview files and transcripts. ## Multilingual datasets ### Multilingual LibriSpeech We used the test splits from each language in [the Multilingual LibriSpeech (MLS) corpus](https://www.openslr.org/94/). ### Fleurs We collected audio files and transcripts using the implementation available as [HuggingFace datasets](https://huggingface.co/datasets/google/fleurs/blob/main/fleurs.py). To use as a translation dataset, we matched the numerical utterance IDs to find the corresponding transcript in English. ### VoxPopuli We used the `get_asr_data.py` script from [the official repository](https://github.com/facebookresearch/voxpopuli) to collect the ASR data in 14 languages. ### Common Voice 9 We downloaded the Common Voice Corpus 9 from [the official website](https://commonvoice.mozilla.org/en/datasets) ### CoVOST 2 We collected the `X into English` data collected using [the official repository](https://github.com/facebookresearch/covost). - INSTRUCTIONS.md (476 bytes)
You are the autonomous runtime coordinator for whisper. System Archetype: tool. Description: Robust Speech Recognition via Large-Scale Weak Supervision Core Directives: 1. Execute operations deterministically following verified repository manifests. 2. Strictly enforce safe input validation and local-first execution boundaries. 3. Provide structured outputs, formatted logs, and actionable resolution steps. 4. Preserve data integrity and respect local workspace isolation.
- LICENSE (1063 bytes)
MIT License Copyright (c) 2022 OpenAI Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
- model-card.md (7307 bytes)
# Model Card: Whisper This is the official codebase for running the automatic speech recognition (ASR) models (Whisper models) trained and released by OpenAI. Following [Model Cards for Model Reporting (Mitchell et al.)](https://arxiv.org/abs/1810.03993), we're providing some information about the automatic speech recognition model. More information on how these models were trained and evaluated can be found [in the paper](https://arxiv.org/abs/2212.04356). ## Model Details The Whisper models are trained for speech recognition and translation tasks, capable of transcribing speech audio into the text in the language it is spoken (ASR) as well as translated into English (speech translation). Researchers at OpenAI developed the models to study the robustness of speech processing systems trained under large-scale weak supervision. There are 9 models of different sizes and capabilities, summarized in the following table. | Size | Parameters | English-only model | Multilingual model | |:------:|:----------:|:------------------:|:------------------:| | tiny | 39 M | ✓ | ✓ | | base | 74 M | ✓ | ✓ | | small | 244 M | ✓ | ✓ | | medium | 769 M | ✓ | ✓ | | large | 1550 M | | ✓ | | turbo | 798 M | | ✓ | In December 2022, we [released an improved large model named `large-v2`](https://github.com/openai/whisper/discussions/661), and `large-v3` in November 2023. Additionally, we've added a `turbo` model in September 2024 which is optimized for inference speed. ### Release date September 2022 (original series), December 2022 (`large-v2`), November 2023 (`large-v3`), September 2024 (`large-v3-turbo`) ### Model type Sequence-to-sequence ASR (automatic speech recognition) and speech translation model ### Paper & samples [Paper](https://arxiv.org/abs/2212.04356) / [Blog](https://openai.com/blog/whisper) ## Model Use ### Evaluated Use The primary intended users of these models are AI researchers studying the robustness, generalization, capabilities, biases, and constraints of the current model. However, Whisper is also potentially quite useful as an ASR solution for developers, especially for English speech recognition. We recognize that once models are released, it is impossible to restrict access to only “intended” uses or to draw reasonable guidelines around what is or is not research. The models are primarily trained and evaluated on ASR and speech translation to English tasks. They show strong ASR results in ~10 languages. They may exhibit additional capabilities, particularly if fine-tuned on certain tasks like voice activity detection, speaker classification, or speaker diarization but have not been robustly evaluated in these areas. We strongly recommend that users perform robust evaluations of the models in a particular context and domain before deploying them. In particular, we caution against using Whisper models to transcribe recordings of individuals taken without their consent or purporting to use these models for any kind of subjective classification. We recommend against use in high-risk domains like decision-making contexts, where flaws in accuracy can lead to pronounced flaws in outcomes. The models are intended to transcribe and translate speech, use of the model for classification is not only not evaluated but also not appropriate, particularly to infer human attributes. ## Training Data The models are trained on 680,000 hours of audio and the corresponding transcripts collected from the internet. 65% of this data (or 438,000 hours) represents English-language audio and matched English transcripts, roughly 18% (or 126,000 hours) represents non-English audio and English transcripts, while the final 17% (or 117,000 hours) represents non-English audio and the corresponding transcript. This non-English data represents 98 different languages. As discussed in [the accompanying paper](https://arxiv.org/abs/2212.04356), we see that performance on transcription in a given language is directly correlated with the amount of training data we employ in that language. ## Performance and Limitations Our studies show that, over many existing ASR systems, the models exhibit improved robustness to accents, background noise, and technical language, as well as zero-shot translation from multiple languages into English; and that accuracy on speech recognition and translation is near the state-of-the-art level. However, because the models are trained in a weakly supervised manner using large-scale noisy data, the predictions may include texts that are not actually spoken in the audio input (i.e. hallucination). We hypothesize that this happens because, given their general knowledge of language, the models combine trying to predict the next word in audio with trying to transcribe the audio itself. Our models perform unevenly across languages, and we observe lower accuracy on low-resource and/or low-discoverability languages or languages where we have less training data. The models also exhibit disparate performance on different accents and dialects of particular languages, which may include a higher word error rate across speakers of different genders, races, ages, or other demographic criteria. Our full evaluation results are presented in [the paper accompanying this release](https://arxiv.org/abs/2212.04356). In addition, the sequence-to-sequence architecture of the model makes it prone to generating repetitive texts, which can be mitigated to some degree by beam search and temperature scheduling but not perfectly. Further analysis of these limitations is provided in [the paper](https://arxiv.org/abs/2212.04356). It is likely that this behavior and hallucinations may be worse in lower-resource and/or lower-discoverability languages. ## Broader Implications We anticipate that Whisper models’ transcription capabilities may be used for improving accessibility tools. While Whisper models cannot be used for real-time transcription out of the box – their speed and size suggest that others may be able to build applications on top of them that allow for near-real-time speech recognition and translation. The real value of beneficial applications built on top of Whisper models suggests that the disparate performance of these models may have real economic implications. There are also potential dual-use concerns that come with releasing Whisper. While we hope the technology will be used primarily for beneficial purposes, making ASR technology more accessible could enable more actors to build capable surveillance technologies or scale up existing surveillance efforts, as the speed and accuracy allow for affordable automatic transcription and translation of large volumes of audio communication. Moreover, these models may have some capabilities to recognize specific individuals out of the box, which in turn presents safety concerns related both to dual use and disparate performance. In practice, we expect that the cost of transcription is not the limiting factor of scaling up surveillance projects.
- pyproject.toml (1418 bytes)
[build-system] build-backend = "setuptools.build_meta" requires = [ "setuptools>=61.2" ] [project] name = "openai-whisper" description = "Robust Speech Recognition via Large-Scale Weak Supervision" readme.content-type = "text/markdown" readme.file = "README.md" license = { text = "MIT" } authors = [ { name = "OpenAI" } ] requires-python = ">=3.8" classifiers = [ "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", ] dynamic = [ "version" ] dependencies = [ "more-itertools", "numba", "numpy", "tiktoken", "torch", "tqdm", "triton>=2; (platform_machine=='x86_64' and sys_platform=='linux') or sys_platform=='linux2'", ] optional-dependencies.dev = [ "black", "flake8", "isort", "pytest", "scipy" ] urls = { Homepage = "https://github.com/openai/whisper" } scripts.whisper = "whisper.transcribe:cli" [tool.setuptools] py-modules = [ "whisper" ] include-package-data = true [tool.setuptools.dynamic] version = { attr = "whisper.version.__version__" } [tool.setuptools.packages.find] exclude = [ "tests*" ] namespaces = false [tool.black] [tool.isort] profile = "black" include_trailing_comma = true line_length = 88 multi_line_output = 3 - README.md (8246 bytes)
# Whisper [[Blog]](https://openai.com/blog/whisper) [[Paper]](https://arxiv.org/abs/2212.04356) [[Model card]](https://github.com/openai/whisper/blob/main/model-card.md) [[Colab example]](https://colab.research.google.com/github/openai/whisper/blob/master/notebooks/LibriSpeech.ipynb) Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. ## Approach  A Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. These tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing a single model to replace many stages of a traditional speech-processing pipeline. The multitask training format uses a set of special tokens that serve as task specifiers or classification targets. ## Setup We used Python 3.9.9 and [PyTorch](https://pytorch.org/) 1.10.1 to train and test our models, but the codebase is expected to be compatible with Python 3.8-3.11 and recent PyTorch versions. The codebase also depends on a few Python packages, most notably [OpenAI's tiktoken](https://github.com/openai/tiktoken) for their fast tokenizer implementation. You can download and install (or update to) the latest release of Whisper with the following command: pip install -U openai-whisper Alternatively, the following command will pull and install the latest commit from this repository, along with its Python dependencies: pip install git+https://github.com/openai/whisper.git To update the package to the latest version of this repository, please run: pip install --upgrade --no-deps --force-reinstall git+https://github.com/openai/whisper.git It also requires the command-line tool [`ffmpeg`](https://ffmpeg.org/) to be installed on your system, which is available from most package managers: ```bash # on Ubuntu or Debian sudo apt update && sudo apt install ffmpeg # on Arch Linux sudo pacman -S ffmpeg # on MacOS using Homebrew (https://brew.sh/) brew install ffmpeg # on Windows using Chocolatey (https://chocolatey.org/) choco install ffmpeg # on Windows using Scoop (https://scoop.sh/) scoop install ffmpeg ``` You may need [`rust`](http://rust-lang.org) installed as well, in case [tiktoken](https://github.com/openai/tiktoken) does not provide a pre-built wheel for your platform. If you see installation errors during the `pip install` command above, please follow the [Getting started page](https://www.rust-lang.org/learn/get-started) to install Rust development environment. Additionally, you may need to configure the `PATH` environment variable, e.g. `export PATH="$HOME/.cargo/bin:$PATH"`. If the installation fails with `No module named 'setuptools_rust'`, you need to install `setuptools_rust`, e.g. by running: ```bash pip install setuptools-rust ``` ## Available models and languages There are six model sizes, four with English-only versions, offering speed and accuracy tradeoffs. Below are the names of the available models and their approximate memory requirements and inference speed relative to the large model. The relative speeds below are measured by transcribing English speech on a A100, and the real-world speed may vary significantly depending on many factors including the language, the speaking speed, and the available hardware. | Size | Parameters | English-only model | Multilingual model | Required VRAM | Relative speed | |:------:|:----------:|:------------------:|:------------------:|:-------------:|:--------------:| | tiny | 39 M | `tiny.en` | `tiny` | ~1 GB | ~10x | | base | 74 M | `base.en` | `base` | ~1 GB | ~7x | | small | 244 M | `small.en` | `small` | ~2 GB | ~4x | | medium | 769 M | `medium.en` | `medium` | ~5 GB | ~2x | | large | 1550 M | N/A | `large` | ~10 GB | 1x | | turbo | 809 M | N/A | `turbo` | ~6 GB | ~8x | The `.en` models for English-only applications tend to perform better, especially for the `tiny.en` and `base.en` models. We observed that the difference becomes less significant for the `small.en` and `medium.en` models. Additionally, the `turbo` model is an optimized version of `large-v3` that offers faster transcription speed with a minimal degradation in accuracy. Whisper's performance varies widely depending on the language. The figure below shows a performance breakdown of `large-v3` and `large-v2` models by language, using WERs (word error rates) or CER (character error rates, shown in *Italic*) evaluated on the Common Voice 15 and Fleurs datasets. Additional WER/CER metrics corresponding to the other models and datasets can be found in Appendix D.1, D.2, and D.4 of [the paper](https://arxiv.org/abs/2212.04356), as well as the BLEU (Bilingual Evaluation Understudy) scores for translation in Appendix D.3.  ## Command-line usage The following command will transcribe speech in audio files, using the `turbo` model: ```bash whisper audio.flac audio.mp3 audio.wav --model turbo ``` The default setting (which selects the `turbo` model) works well for transcribing English. However, **the `turbo` model is not trained for translation tasks**. If you need to **translate non-English speech into English**, use one of the **multilingual models** (`tiny`, `base`, `small`, `medium`, `large`) instead of `turbo`. For example, to transcribe an audio file containing non-English speech, you can specify the language: ```bash whisper japanese.wav --language Japanese ``` To **translate** speech into English, use: ```bash whisper japanese.wav --model medium --language Japanese --task translate ``` > **Note:** The `turbo` model will return the original language even if `--task translate` is specified. Use `medium` or `large` for the best translation results. Run the following to view all available options: ```bash whisper --help ``` See [tokenizer.py](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py) for the list of all available languages. ## Python usage Transcription can also be performed within Python: ```python import whisper model = whisper.load_model("turbo") result = model.transcribe("audio.mp3") print(result["text"]) ``` Internally, the `transcribe()` method reads the entire file and processes the audio with a sliding 30-second window, performing autoregressive sequence-to-sequence predictions on each window. Below is an example usage of `whisper.detect_language()` and `whisper.decode()` which provide lower-level access to the model. ```python import whisper model = whisper.load_model("turbo") # load audio and pad/trim it to fit 30 seconds audio = whisper.load_audio("audio.mp3") audio = whisper.pad_or_trim(audio) # make log-Mel spectrogram and move to the same device as the model mel = whisper.log_mel_spectrogram(audio, n_mels=model.dims.n_mels).to(model.device) # detect the spoken language _, probs = model.detect_language(mel) print(f"Detected language: {max(probs, key=probs.get)}") # decode the audio options = whisper.DecodingOptions() result = whisper.decode(model, mel, options) # print the recognized text print(result.text) ``` ## More examples Please use the [🙌 Show and tell](https://github.com/openai/whisper/discussions/categories/show-and-tell) category in Discussions for sharing more example usages of Whisper and third-party extensions such as web demos, integrations with other tools, ports for different platforms, etc. ## License Whisper's code and model weights are released under the MIT License. See [LICENSE](https://github.com/openai/whisper/blob/main/LICENSE) for further details. - requirements.txt (140 bytes)
numba numpy torch tqdm more-itertools tiktoken triton>=2.0.0;platform_machine=="x86_64" and sys_platform=="linux" or sys_platform=="linux2"
- whisper/__init__.py (7432 bytes)
import hashlib import io import os import urllib import warnings from typing import List, Optional, Union import torch from tqdm import tqdm from .audio import load_audio, log_mel_spectrogram, pad_or_trim from .decoding import DecodingOptions, DecodingResult, decode, detect_language from .model import ModelDimensions, Whisper from .transcribe import transcribe from .version import __version__ _MODELS = { "tiny.en": "https://openaipublic.azureedge.net/main/whisper/models/d3dd57d32accea0b295c96e26691aa14d8822fac7d9d27d5dc00b4ca2826dd03/tiny.en.pt", "tiny": "https://openaipublic.azureedge.net/main/whisper/models/65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9/tiny.pt", "base.en": "https://openaipublic.azureedge.net/main/whisper/models/25a8566e1d0c1e2231d1c762132cd20e0f96a85d16145c3a00adf5d1ac670ead/base.en.pt", "base": "https://openaipublic.azureedge.net/main/whisper/models/ed3a0b6b1c0edf879ad9b11b1af5a0e6ab5db9205f891f668f8b0e6c6326e34e/base.pt", "small.en": "https://openaipublic.azureedge.net/main/whisper/models/f953ad0fd29cacd07d5a9eda5624af0f6bcf2258be67c92b79389873d91e0872/small.en.pt", "small": "https://openaipublic.azureedge.net/main/whisper/models/9ecf779972d90ba49c06d968637d720dd632c55bbf19d441fb42bf17a411e794/small.pt", "medium.en": "https://openaipublic.azureedge.net/main/whisper/models/d7440d1dc186f76616474e0ff0b3b6b879abc9d1a4926b7adfa41db2d497ab4f/medium.en.pt", "medium": "https://openaipublic.azureedge.net/main/whisper/models/345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1/medium.pt", "large-v1": "https://openaipublic.azureedge.net/main/whisper/models/e4b87e7e0bf463eb8e6956e646f1e277e901512310def2c24bf0e11bd3c28e9a/large-v1.pt", "large-v2": "https://openaipublic.azureedge.net/main/whisper/models/81f7c96c852ee8fc832187b0132e569d6c3065a3252ed18e56effd0b6a73e524/large-v2.pt", "large-v3": "https://openaipublic.azureedge.net/main/whisper/models/e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/large-v3.pt", "large": "https://openaipublic.azureedge.net/main/whisper/models/e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/large-v3.pt", "large-v3-turbo": "https://openaipublic.azureedge.net/main/whisper/models/aff26ae408abcba5fbf8813c21e62b0941638c5f6eebfb145be0c9839262a19a/large-v3-turbo.pt", "turbo": "https://openaipublic.azureedge.net/main/whisper/models/aff26ae408abcba5fbf8813c21e62b0941638c5f6eebfb145be0c9839262a19a/large-v3-turbo.pt", } # base85-encoded (n_layers, n_heads) boolean arrays indicating the cross-attention heads that are # highly correlated to the word-level timing, i.e. the alignment between audio and text tokens. _ALIGNMENT_HEADS = { "tiny.en": b"ABzY8J1N>@0{>%R00Bk>$p{7v037`oCl~+#00", "tiny": b"ABzY8bu8Lr0{>%RKn9Fp%m@SkK7Kt=7ytkO", "base.en": b"ABzY8;40c<0{>%RzzG;p*o+Vo09|#PsxSZm00", "base": b"ABzY8KQ!870{>%RzyTQH3`Q^yNP!>##QT-<FaQ7m", "small.en": b"ABzY8>?_)10{>%RpeA61k&I|OI3I$65C{;;pbCHh0B{qLQ;+}v00", "small": b"ABzY8DmU6=0{>%Rpa?J`kvJ6qF(V^F86#Xh7JUGMK}P<N0000", "medium.en": b"ABzY8usPae0{>%R7<zz_OvQ{)4kMa0BMw6u5rT}kRKX;$NfYBv00*Hl@qhsU00", "medium": b"ABzY8B0Jh+0{>%R7}kK1fFL7w6%<-Pf*t^=N)Qr&0RR9", "large-v1": b"ABzY8r9j$a0{>%R7#4sLmoOs{s)o3~84-RPdcFk!JR<kSfC2yj", "large-v2": b"ABzY8zd+h!0{>%R7=D0pU<_bnWW*tkYAhobTNnu$jnkEkXqp)j;w1Tzk)UH3X%SZd&fFZ2fC2yj", "large-v3": b"ABzY8gWO1E0{>%R7(9S+Kn!D~%ngiGaR?*L!iJG9p-nab0JQ=-{D1-g00", "large": b"ABzY8gWO1E0{>%R7(9S+Kn!D~%ngiGaR?*L!iJG9p-nab0JQ=-{D1-g00", "large-v3-turbo": b"ABzY8j^C+e0{>%RARaKHP%t(lGR*)0g!tONPyhe`", "turbo": b"ABzY8j^C+e0{>%RARaKHP%t(lGR*)0g!tONPyhe`", } def _download(url: str, root: str, in_memory: bool) -> Union[bytes, str]: os.makedirs(root, exist_ok=True) expected_sha256 = url.split("/")[-2] download_target = os.path.join(root, os.path.basename(url)) if os.path.exists(download_target) and not os.path.isfile(download_target): raise RuntimeError(f"{download_target} exists and is not a regular file") if os.path.isfile(download_target): with open(download_target, "rb") as f: model_bytes = f.read() if hashlib.sha256(model_bytes).hexdigest() == expected_sha256: return model_bytes if in_memory else download_target else: warnings.warn( f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file" ) with urllib.request.urlopen(url) as source, open(download_target, "wb") as output: with tqdm( total=int(source.info().get("Content-Length")), ncols=80, unit="iB", unit_scale=True, unit_divisor=1024, ) as loop: while True: buffer = source.read(8192) if not buffer: break output.write(buffer) loop.update(len(buffer)) model_bytes = open(download_target, "rb").read() if hashlib.sha256(model_bytes).hexdigest() != expected_sha256: raise RuntimeError( "Model has been downloaded but the SHA256 checksum does not not match. Please retry loading the model." ) return model_bytes if in_memory else download_target def available_models() -> List[str]: """Returns the names of available models""" return list(_MODELS.keys()) def load_model( name: str, device: Optional[Union[str, torch.device]] = None, download_root: str = None, in_memory: bool = False, ) -> Whisper: """ Load a Whisper ASR model Parameters ---------- name : str one of the official model names listed by `whisper.available_models()`, or path to a model checkpoint containing the model dimensions and the model state_dict. device : Union[str, torch.device] the PyTorch device to put the model into download_root: str path to download the model files; by default, it uses "~/.cache/whisper" in_memory: bool whether to preload the model weights into host memory Returns ------- model : Whisper The Whisper ASR model instance """ if device is None: device = "cuda" if torch.cuda.is_available() else "cpu" if download_root is None: default = os.path.join(os.path.expanduser("~"), ".cache") download_root = os.path.join(os.getenv("XDG_CACHE_HOME", default), "whisper") if name in _MODELS: checkpoint_file = _download(_MODELS[name], download_root, in_memory) alignment_heads = _ALIGNMENT_HEADS[name] elif os.path.isfile(name): checkpoint_file = open(name, "rb").read() if in_memory else name alignment_heads = None else: raise RuntimeError( f"Model {name} not found; available models = {available_models()}" ) with ( io.BytesIO(checkpoint_file) if in_memory else open(checkpoint_file, "rb") ) as fp: kwargs = {"weights_only": True} if torch.__version__ >= "1.13" else {} checkpoint = torch.load(fp, map_location=device, **kwargs) del checkpoint_file dims = ModelDimensions(**checkpoint["dims"]) model = Whisper(dims) model.load_state_dict(checkpoint["model_state_dict"]) if alignment_heads is not None: model.set_alignment_heads(alignment_heads) return model.to(device) - whisper/__main__.py (35 bytes)
from .transcribe import cli cli()
- whisper/audio.py (4945 bytes)
import os from functools import lru_cache from subprocess import CalledProcessError, run from typing import Optional, Union import numpy as np import torch import torch.nn.functional as F from .utils import exact_div # hard-coded audio hyperparameters SAMPLE_RATE = 16000 N_FFT = 400 HOP_LENGTH = 160 CHUNK_LENGTH = 30 N_SAMPLES = CHUNK_LENGTH * SAMPLE_RATE # 480000 samples in a 30-second chunk N_FRAMES = exact_div(N_SAMPLES, HOP_LENGTH) # 3000 frames in a mel spectrogram input N_SAMPLES_PER_TOKEN = HOP_LENGTH * 2 # the initial convolutions has stride 2 FRAMES_PER_SECOND = exact_div(SAMPLE_RATE, HOP_LENGTH) # 10ms per audio frame TOKENS_PER_SECOND = exact_div(SAMPLE_RATE, N_SAMPLES_PER_TOKEN) # 20ms per audio token def load_audio(file: str, sr: int = SAMPLE_RATE): """ Open an audio file and read as mono waveform, resampling as necessary Parameters ---------- file: str The audio file to open sr: int The sample rate to resample the audio if necessary Returns ------- A NumPy array containing the audio waveform, in float32 dtype. """ # This launches a subprocess to decode audio while down-mixing # and resampling as necessary. Requires the ffmpeg CLI in PATH. # fmt: off cmd = [ "ffmpeg", "-nostdin", "-threads", "0", "-i", file, "-f", "s16le", "-ac", "1", "-acodec", "pcm_s16le", "-ar", str(sr), "-" ] # fmt: on try: out = run(cmd, capture_output=True, check=True).stdout except CalledProcessError as e: raise RuntimeError(f"Failed to load audio: {e.stderr.decode()}") from e return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0 def pad_or_trim(array, length: int = N_SAMPLES, *, axis: int = -1): """ Pad or trim the audio array to N_SAMPLES, as expected by the encoder. """ if torch.is_tensor(array): if array.shape[axis] > length: array = array.index_select( dim=axis, index=torch.arange(length, device=array.device) ) if array.shape[axis] < length: pad_widths = [(0, 0)] * array.ndim pad_widths[axis] = (0, length - array.shape[axis]) array = F.pad(array, [pad for sizes in pad_widths[::-1] for pad in sizes]) else: if array.shape[axis] > length: array = array.take(indices=range(length), axis=axis) if array.shape[axis] < length: pad_widths = [(0, 0)] * array.ndim pad_widths[axis] = (0, length - array.shape[axis]) array = np.pad(array, pad_widths) return array @lru_cache(maxsize=None) def mel_filters(device, n_mels: int) -> torch.Tensor: """ load the mel filterbank matrix for projecting STFT into a Mel spectrogram. Allows decoupling librosa dependency; saved using: np.savez_compressed( "mel_filters.npz", mel_80=librosa.filters.mel(sr=16000, n_fft=400, n_mels=80), mel_128=librosa.filters.mel(sr=16000, n_fft=400, n_mels=128), ) """ assert n_mels in {80, 128}, f"Unsupported n_mels: {n_mels}" filters_path = os.path.join(os.path.dirname(__file__), "assets", "mel_filters.npz") with np.load(filters_path, allow_pickle=False) as f: return torch.from_numpy(f[f"mel_{n_mels}"]).to(device) def log_mel_spectrogram( audio: Union[str, np.ndarray, torch.Tensor], n_mels: int = 80, padding: int = 0, device: Optional[Union[str, torch.device]] = None, ): """ Compute the log-Mel spectrogram of Parameters ---------- audio: Union[str, np.ndarray, torch.Tensor], shape = (*) The path to audio or either a NumPy array or Tensor containing the audio waveform in 16 kHz n_mels: int The number of Mel-frequency filters, only 80 and 128 are supported padding: int Number of zero samples to pad to the right device: Optional[Union[str, torch.device]] If given, the audio tensor is moved to this device before STFT Returns ------- torch.Tensor, shape = (n_mels, n_frames) A Tensor that contains the Mel spectrogram """ if not torch.is_tensor(audio): if isinstance(audio, str): audio = load_audio(audio) audio = torch.from_numpy(audio) if device is not None: audio = audio.to(device) if padding > 0: audio = F.pad(audio, (0, padding)) window = torch.hann_window(N_FFT).to(audio.device) stft = torch.stft(audio, N_FFT, HOP_LENGTH, window=window, return_complex=True) magnitudes = stft[..., :-1].abs() ** 2 filters = mel_filters(audio.device, n_mels) mel_spec = filters @ magnitudes log_spec = torch.clamp(mel_spec, min=1e-10).log10() log_spec = torch.maximum(log_spec, log_spec.max() - 8.0) log_spec = (log_spec + 4.0) / 4.0 return log_spec - whisper/decoding.py (32155 bytes)
from dataclasses import dataclass, field, replace from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Sequence, Tuple, Union import numpy as np import torch import torch.nn.functional as F from torch import Tensor from torch.distributions import Categorical from .audio import CHUNK_LENGTH from .tokenizer import Tokenizer, get_tokenizer from .utils import compression_ratio if TYPE_CHECKING: from .model import Whisper @torch.no_grad() def detect_language( model: "Whisper", mel: Tensor, tokenizer: Tokenizer = None ) -> Tuple[Tensor, List[dict]]: """ Detect the spoken language in the audio, and return them as list of strings, along with the ids of the most probable language tokens and the probability distribution over all language tokens. This is performed outside the main decode loop in order to not interfere with kv-caching. Returns ------- language_tokens : Tensor, shape = (n_audio,) ids of the most probable language tokens, which appears after the startoftranscript token. language_probs : List[Dict[str, float]], length = n_audio list of dictionaries containing the probability distribution over all languages. """ if tokenizer is None: tokenizer = get_tokenizer( model.is_multilingual, num_languages=model.num_languages ) if ( tokenizer.language is None or tokenizer.language_token not in tokenizer.sot_sequence ): raise ValueError( "This model doesn't have language tokens so it can't perform lang id" ) single = mel.ndim == 2 if single: mel = mel.unsqueeze(0) # skip encoder forward pass if already-encoded audio features were given if mel.shape[-2:] != (model.dims.n_audio_ctx, model.dims.n_audio_state): mel = model.encoder(mel) # forward pass using a single token, startoftranscript n_audio = mel.shape[0] x = torch.tensor([[tokenizer.sot]] * n_audio).to(mel.device) # [n_audio, 1] logits = model.logits(x, mel)[:, 0] # collect detected languages; suppress all non-language tokens mask = torch.ones(logits.shape[-1], dtype=torch.bool) mask[list(tokenizer.all_language_tokens)] = False logits[:, mask] = -np.inf language_tokens = logits.argmax(dim=-1) language_token_probs = logits.softmax(dim=-1).cpu() language_probs = [ { c: language_token_probs[i, j].item() for j, c in zip(tokenizer.all_language_tokens, tokenizer.all_language_codes) } for i in range(n_audio) ] if single: language_tokens = language_tokens[0] language_probs = language_probs[0] return language_tokens, language_probs @dataclass(frozen=True) class DecodingOptions: # whether to perform X->X "transcribe" or X->English "translate" task: str = "transcribe" # language that the audio is in; uses detected language if None language: Optional[str] = None # sampling-related options temperature: float = 0.0 sample_len: Optional[int] = None # maximum number of tokens to sample best_of: Optional[int] = None # number of independent sample trajectories, if t > 0 beam_size: Optional[int] = None # number of beams in beam search, if t == 0 patience: Optional[float] = None # patience in beam search (arxiv:2204.05424) # "alpha" in Google NMT, or None for length norm, when ranking generations # to select which to return among the beams or best-of-N samples length_penalty: Optional[float] = None # text or tokens to feed as the prompt or the prefix; for more info: # https://github.com/openai/whisper/discussions/117#discussioncomment-3727051 prompt: Optional[Union[str, List[int]]] = None # for the previous context prefix: Optional[Union[str, List[int]]] = None # to prefix the current context # list of tokens ids (or comma-separated token ids) to suppress # "-1" will suppress a set of symbols as defined in `tokenizer.non_speech_tokens()` suppress_tokens: Optional[Union[str, Iterable[int]]] = "-1" suppress_blank: bool = True # this will suppress blank outputs # timestamp sampling options without_timestamps: bool = False # use <|notimestamps|> to sample text tokens only max_initial_timestamp: Optional[float] = 1.0 # implementation details fp16: bool = True # use fp16 for most of the calculation @dataclass(frozen=True) class DecodingResult: audio_features: Tensor language: str language_probs: Optional[Dict[str, float]] = None tokens: List[int] = field(default_factory=list) text: str = "" avg_logprob: float = np.nan no_speech_prob: float = np.nan temperature: float = np.nan compression_ratio: float = np.nan class Inference: def logits(self, tokens: Tensor, audio_features: Tensor) -> Tensor: """Perform a forward pass on the decoder and return per-token logits""" raise NotImplementedError def rearrange_kv_cache(self, source_indices) -> None: """Update the key-value cache according to the updated beams""" raise NotImplementedError def cleanup_caching(self) -> None: """Clean up any resources or hooks after decoding is finished""" pass class PyTorchInference(Inference): def __init__(self, model: "Whisper", initial_token_length: int): self.model: "Whisper" = model self.initial_token_length = initial_token_length self.kv_cache = {} self.hooks = [] key_modules = [block.attn.key for block in self.model.decoder.blocks] value_modules = [block.attn.value for block in self.model.decoder.blocks] self.kv_modules = key_modules + value_modules def logits(self, tokens: Tensor, audio_features: Tensor) -> Tensor: if not self.kv_cache: self.kv_cache, self.hooks = self.model.install_kv_cache_hooks() if tokens.shape[-1] > self.initial_token_length: # only need to use the last token except in the first forward pass tokens = tokens[:, -1:] return self.model.decoder(tokens, audio_features, kv_cache=self.kv_cache) def cleanup_caching(self): for hook in self.hooks: hook.remove() self.kv_cache = {} self.hooks = [] def rearrange_kv_cache(self, source_indices): if source_indices != list(range(len(source_indices))): for module in self.kv_modules: # update the key/value cache to contain the selected sequences self.kv_cache[module] = self.kv_cache[module][source_indices].detach() class SequenceRanker: def rank( self, tokens: List[List[Tensor]], sum_logprobs: List[List[float]] ) -> List[int]: """ Given a list of groups of samples and their cumulative log probabilities, return the indices of the samples in each group to select as the final result """ raise NotImplementedError class MaximumLikelihoodRanker(SequenceRanker): """ Select the sample with the highest log probabilities, penalized using either a simple length normalization or Google NMT paper's length penalty """ def __init__(self, length_penalty: Optional[float]): self.length_penalty = length_penalty def rank(self, tokens: List[List[Tensor]], sum_logprobs: List[List[float]]): def scores(logprobs, lengths): result = [] for logprob, length in zip(logprobs, lengths): if self.length_penalty is None: penalty = length else: # from the Google NMT paper penalty = ((5 + length) / 6) ** self.length_penalty result.append(logprob / penalty) return result # get the sequence with the highest score lengths = [[len(t) for t in s] for s in tokens] return [np.argmax(scores(p, l)) for p, l in zip(sum_logprobs, lengths)] class TokenDecoder: def reset(self): """Initialize any stateful variables for decoding a new sequence""" def update( self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor ) -> Tuple[Tensor, bool]: """Specify how to select the next token, based on the current trace and logits Parameters ---------- tokens : Tensor, shape = (n_batch, current_sequence_length) all tokens in the context so far, including the prefix and sot_sequence tokens logits : Tensor, shape = (n_batch, vocab_size) per-token logits of the probability distribution at the current step sum_logprobs : Tensor, shape = (n_batch) cumulative log probabilities for each sequence Returns ------- tokens : Tensor, shape = (n_batch, current_sequence_length + 1) the tokens, appended with the selected next token completed : bool True if all sequences has reached the end of text """ raise NotImplementedError def finalize( self, tokens: Tensor, sum_logprobs: Tensor ) -> Tuple[Sequence[Sequence[Tensor]], List[List[float]]]: """Finalize search and return the final candidate sequences Parameters ---------- tokens : Tensor, shape = (n_audio, n_group, current_sequence_length) all tokens in the context so far, including the prefix and sot_sequence sum_logprobs : Tensor, shape = (n_audio, n_group) cumulative log probabilities for each sequence Returns ------- tokens : Sequence[Sequence[Tensor]], length = n_audio sequence of Tensors containing candidate token sequences, for each audio input sum_logprobs : List[List[float]], length = n_audio sequence of cumulative log probabilities corresponding to the above """ raise NotImplementedError class GreedyDecoder(TokenDecoder): def __init__(self, temperature: float, eot: int): self.temperature = temperature self.eot = eot def update( self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor ) -> Tuple[Tensor, bool]: if self.temperature == 0: next_tokens = logits.argmax(dim=-1) else: next_tokens = Categorical(logits=logits / self.temperature).sample() logprobs = F.log_softmax(logits.float(), dim=-1) current_logprobs = logprobs[torch.arange(logprobs.shape[0]), next_tokens] sum_logprobs += current_logprobs * (tokens[:, -1] != self.eot) next_tokens[tokens[:, -1] == self.eot] = self.eot tokens = torch.cat([tokens, next_tokens[:, None]], dim=-1) completed = (tokens[:, -1] == self.eot).all() return tokens, completed def finalize(self, tokens: Tensor, sum_logprobs: Tensor): # make sure each sequence has at least one EOT token at the end tokens = F.pad(tokens, (0, 1), value=self.eot) return tokens, sum_logprobs.tolist() class BeamSearchDecoder(TokenDecoder): def __init__( self, beam_size: int, eot: int, inference: Inference, patience: Optional[float] = None, ): self.beam_size = beam_size self.eot = eot self.inference = inference self.patience = patience or 1.0 self.max_candidates: int = round(beam_size * self.patience) self.finished_sequences = None assert ( self.max_candidates > 0 ), f"Invalid beam size ({beam_size}) or patience ({patience})" def reset(self): self.finished_sequences = None def update( self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor ) -> Tuple[Tensor, bool]: if tokens.shape[0] % self.beam_size != 0: raise ValueError(f"{tokens.shape}[0] % {self.beam_size} != 0") n_audio = tokens.shape[0] // self.beam_size if self.finished_sequences is None: # for the first update self.finished_sequences = [{} for _ in range(n_audio)] logprobs = F.log_softmax(logits.float(), dim=-1) next_tokens, source_indices, finished_sequences = [], [], [] for i in range(n_audio): scores, sources, finished = {}, {}, {} # STEP 1: calculate the cumulative log probabilities for possible candidates for j in range(self.beam_size): idx = i * self.beam_size + j prefix = tokens[idx].tolist() for logprob, token in zip(*logprobs[idx].topk(self.beam_size + 1)): new_logprob = (sum_logprobs[idx] + logprob).item() sequence = tuple(prefix + [token.item()]) scores[sequence] = new_logprob sources[sequence] = idx # STEP 2: rank the candidates and keep the top beam_size sequences for each audio saved = 0 for sequence in sorted(scores, key=scores.get, reverse=True): if sequence[-1] == self.eot: finished[sequence] = scores[sequence] else: sum_logprobs[len(next_tokens)] = scores[sequence] next_tokens.append(sequence) source_indices.append(sources[sequence]) saved += 1 if saved == self.beam_size: break finished_sequences.append(finished) tokens = torch.tensor(next_tokens, device=tokens.device) self.inference.rearrange_kv_cache(source_indices) # add newly finished sequences to self.finished_sequences assert len(self.finished_sequences) == len(finished_sequences) for previously_finished, newly_finished in zip( self.finished_sequences, finished_sequences ): for seq in sorted(newly_finished, key=newly_finished.get, reverse=True): if len(previously_finished) >= self.max_candidates: break # the candidate list is full previously_finished[seq] = newly_finished[seq] # mark as completed if all audio has enough number of samples completed = all( len(sequences) >= self.max_candidates for sequences in self.finished_sequences ) return tokens, completed def finalize(self, preceding_tokens: Tensor, sum_logprobs: Tensor): # collect all finished sequences, including patience, and add unfinished ones if not enough sum_logprobs = sum_logprobs.cpu() for i, sequences in enumerate(self.finished_sequences): if ( len(sequences) < self.beam_size ): # when not enough sequences are finished for j in list(np.argsort(sum_logprobs[i]))[::-1]: sequence = preceding_tokens[i, j].tolist() + [self.eot] se - whisper/model.py (12209 bytes)
import base64 import gzip from contextlib import contextmanager from dataclasses import dataclass from typing import Dict, Iterable, Optional, Tuple import numpy as np import torch import torch.nn.functional as F from torch import Tensor, nn from .decoding import decode as decode_function from .decoding import detect_language as detect_language_function from .transcribe import transcribe as transcribe_function try: from torch.nn.functional import scaled_dot_product_attention SDPA_AVAILABLE = True except (ImportError, RuntimeError, OSError): scaled_dot_product_attention = None SDPA_AVAILABLE = False @dataclass class ModelDimensions: n_mels: int n_audio_ctx: int n_audio_state: int n_audio_head: int n_audio_layer: int n_vocab: int n_text_ctx: int n_text_state: int n_text_head: int n_text_layer: int class LayerNorm(nn.LayerNorm): def forward(self, x: Tensor) -> Tensor: return super().forward(x.float()).type(x.dtype) class Linear(nn.Linear): def forward(self, x: Tensor) -> Tensor: return F.linear( x, self.weight.to(x.dtype), None if self.bias is None else self.bias.to(x.dtype), ) class Conv1d(nn.Conv1d): def _conv_forward( self, x: Tensor, weight: Tensor, bias: Optional[Tensor] ) -> Tensor: return super()._conv_forward( x, weight.to(x.dtype), None if bias is None else bias.to(x.dtype) ) def sinusoids(length, channels, max_timescale=10000): """Returns sinusoids for positional embedding""" assert channels % 2 == 0 log_timescale_increment = np.log(max_timescale) / (channels // 2 - 1) inv_timescales = torch.exp(-log_timescale_increment * torch.arange(channels // 2)) scaled_time = torch.arange(length)[:, np.newaxis] * inv_timescales[np.newaxis, :] return torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], dim=1) @contextmanager def disable_sdpa(): prev_state = MultiHeadAttention.use_sdpa try: MultiHeadAttention.use_sdpa = False yield finally: MultiHeadAttention.use_sdpa = prev_state class MultiHeadAttention(nn.Module): use_sdpa = True def __init__(self, n_state: int, n_head: int): super().__init__() self.n_head = n_head self.query = Linear(n_state, n_state) self.key = Linear(n_state, n_state, bias=False) self.value = Linear(n_state, n_state) self.out = Linear(n_state, n_state) def forward( self, x: Tensor, xa: Optional[Tensor] = None, mask: Optional[Tensor] = None, kv_cache: Optional[dict] = None, ): q = self.query(x) if kv_cache is None or xa is None or self.key not in kv_cache: # hooks, if installed (i.e. kv_cache is not None), will prepend the cached kv tensors; # otherwise, perform key/value projections for self- or cross-attention as usual. k = self.key(x if xa is None else xa) v = self.value(x if xa is None else xa) else: # for cross-attention, calculate keys and values once and reuse in subsequent calls. k = kv_cache[self.key] v = kv_cache[self.value] wv, qk = self.qkv_attention(q, k, v, mask) return self.out(wv), qk def qkv_attention( self, q: Tensor, k: Tensor, v: Tensor, mask: Optional[Tensor] = None ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]: n_batch, n_ctx, n_state = q.shape scale = (n_state // self.n_head) ** -0.25 q = q.view(*q.shape[:2], self.n_head, -1).permute(0, 2, 1, 3) k = k.view(*k.shape[:2], self.n_head, -1).permute(0, 2, 1, 3) v = v.view(*v.shape[:2], self.n_head, -1).permute(0, 2, 1, 3) if SDPA_AVAILABLE and MultiHeadAttention.use_sdpa: if k.shape[0] == 1 and q.shape[0] != 1: # Cross-attention K/V have batch 1 and broadcast against the # beam-expanded query; the fused SDPA kernels reject the batch # mismatch and fall back to math, so expand K/V to a stride-0 # view (no copy) to keep them on the fast path. k = k.expand(q.shape[0], *k.shape[1:]) v = v.expand(q.shape[0], *v.shape[1:]) a = scaled_dot_product_attention( q, k, v, is_causal=mask is not None and n_ctx > 1 ) out = a.permute(0, 2, 1, 3).flatten(start_dim=2) qk = None else: qk = (q * scale) @ (k * scale).transpose(-1, -2) if mask is not None: qk = qk + mask[:n_ctx, :n_ctx] qk = qk.float() w = F.softmax(qk, dim=-1).to(q.dtype) out = (w @ v).permute(0, 2, 1, 3).flatten(start_dim=2) qk = qk.detach() return out, qk class ResidualAttentionBlock(nn.Module): def __init__(self, n_state: int, n_head: int, cross_attention: bool = False): super().__init__() self.attn = MultiHeadAttention(n_state, n_head) self.attn_ln = LayerNorm(n_state) self.cross_attn = ( MultiHeadAttention(n_state, n_head) if cross_attention else None ) self.cross_attn_ln = LayerNorm(n_state) if cross_attention else None n_mlp = n_state * 4 self.mlp = nn.Sequential( Linear(n_state, n_mlp), nn.GELU(), Linear(n_mlp, n_state) ) self.mlp_ln = LayerNorm(n_state) def forward( self, x: Tensor, xa: Optional[Tensor] = None, mask: Optional[Tensor] = None, kv_cache: Optional[dict] = None, ): x = x + self.attn(self.attn_ln(x), mask=mask, kv_cache=kv_cache)[0] if self.cross_attn: x = x + self.cross_attn(self.cross_attn_ln(x), xa, kv_cache=kv_cache)[0] x = x + self.mlp(self.mlp_ln(x)) return x class AudioEncoder(nn.Module): def __init__( self, n_mels: int, n_ctx: int, n_state: int, n_head: int, n_layer: int ): super().__init__() self.conv1 = Conv1d(n_mels, n_state, kernel_size=3, padding=1) self.conv2 = Conv1d(n_state, n_state, kernel_size=3, stride=2, padding=1) self.register_buffer("positional_embedding", sinusoids(n_ctx, n_state)) self.blocks: Iterable[ResidualAttentionBlock] = nn.ModuleList( [ResidualAttentionBlock(n_state, n_head) for _ in range(n_layer)] ) self.ln_post = LayerNorm(n_state) def forward(self, x: Tensor): """ x : torch.Tensor, shape = (batch_size, n_mels, n_ctx) the mel spectrogram of the audio """ x = F.gelu(self.conv1(x)) x = F.gelu(self.conv2(x)) x = x.permute(0, 2, 1) assert x.shape[1:] == self.positional_embedding.shape, "incorrect audio shape" x = (x + self.positional_embedding).to(x.dtype) for block in self.blocks: x = block(x) x = self.ln_post(x) return x class TextDecoder(nn.Module): def __init__( self, n_vocab: int, n_ctx: int, n_state: int, n_head: int, n_layer: int ): super().__init__() self.token_embedding = nn.Embedding(n_vocab, n_state) self.positional_embedding = nn.Parameter(torch.empty(n_ctx, n_state)) self.blocks: Iterable[ResidualAttentionBlock] = nn.ModuleList( [ ResidualAttentionBlock(n_state, n_head, cross_attention=True) for _ in range(n_layer) ] ) self.ln = LayerNorm(n_state) mask = torch.empty(n_ctx, n_ctx).fill_(-np.inf).triu_(1) self.register_buffer("mask", mask, persistent=False) def forward(self, x: Tensor, xa: Tensor, kv_cache: Optional[dict] = None): """ x : torch.LongTensor, shape = (batch_size, <= n_ctx) the text tokens xa : torch.Tensor, shape = (batch_size, n_audio_ctx, n_audio_state) the encoded audio features to be attended on """ offset = next(iter(kv_cache.values())).shape[1] if kv_cache else 0 x = ( self.token_embedding(x) + self.positional_embedding[offset : offset + x.shape[-1]] ) x = x.to(xa.dtype) for block in self.blocks: x = block(x, xa, mask=self.mask, kv_cache=kv_cache) x = self.ln(x) logits = ( x @ torch.transpose(self.token_embedding.weight.to(x.dtype), 0, 1) ).float() return logits class Whisper(nn.Module): def __init__(self, dims: ModelDimensions): super().__init__() self.dims = dims self.encoder = AudioEncoder( self.dims.n_mels, self.dims.n_audio_ctx, self.dims.n_audio_state, self.dims.n_audio_head, self.dims.n_audio_layer, ) self.decoder = TextDecoder( self.dims.n_vocab, self.dims.n_text_ctx, self.dims.n_text_state, self.dims.n_text_head, self.dims.n_text_layer, ) # use the last half among the decoder layers for time alignment by default; # to use a specific set of heads, see `set_alignment_heads()` below. all_heads = torch.zeros( self.dims.n_text_layer, self.dims.n_text_head, dtype=torch.bool ) all_heads[self.dims.n_text_layer // 2 :] = True self.register_buffer("alignment_heads", all_heads.to_sparse(), persistent=False) def set_alignment_heads(self, dump: bytes): array = np.frombuffer( gzip.decompress(base64.b85decode(dump)), dtype=bool ).copy() mask = torch.from_numpy(array).reshape( self.dims.n_text_layer, self.dims.n_text_head ) self.register_buffer("alignment_heads", mask.to_sparse(), persistent=False) def embed_audio(self, mel: torch.Tensor): return self.encoder(mel) def logits(self, tokens: torch.Tensor, audio_features: torch.Tensor): return self.decoder(tokens, audio_features) def forward( self, mel: torch.Tensor, tokens: torch.Tensor ) -> Dict[str, torch.Tensor]: return self.decoder(tokens, self.encoder(mel)) @property def device(self): return next(self.parameters()).device @property def is_multilingual(self): return self.dims.n_vocab >= 51865 @property def num_languages(self): return self.dims.n_vocab - 51765 - int(self.is_multilingual) def install_kv_cache_hooks(self, cache: Optional[dict] = None): """ The `MultiHeadAttention` module optionally accepts `kv_cache` which stores the key and value tensors calculated for the previous positions. This method returns a dictionary that stores all caches, and the necessary hooks for the key and value projection modules that save the intermediate tensors to be reused during later calculations. Returns ------- cache : Dict[nn.Module, torch.Tensor] A dictionary object mapping the key/value projection modules to its cache hooks : List[RemovableHandle] List of PyTorch RemovableHandle objects to stop the hooks to be called """ cache = {**cache} if cache is not None else {} hooks = [] def save_to_cache(module, _, output): if module not in cache or output.shape[1] > self.dims.n_text_ctx: # save as-is, for the first token or cross attention cache[module] = output else: cache[module] = torch.cat([cache[module], output], dim=1).detach() return cache[module] def install_hooks(layer: nn.Module): if isinstance(layer, MultiHeadAttention): hooks.append(layer.key.register_forward_hook(save_to_cache)) hooks.append(layer.value.register_forward_hook(save_to_cache)) self.decoder.apply(install_hooks) return cache, hooks detect_language = detect_language_function transcribe = transcribe_function decode = decode_function - whisper/normalizers/__init__.py (130 bytes)
from .basic import BasicTextNormalizer as BasicTextNormalizer from .english import EnglishTextNormalizer as EnglishTextNormalizer
- whisper/normalizers/basic.py (2064 bytes)
import re import unicodedata import regex # non-ASCII letters that are not separated by "NFKD" normalization ADDITIONAL_DIACRITICS = { "œ": "oe", "Œ": "OE", "ø": "o", "Ø": "O", "æ": "ae", "Æ": "AE", "ß": "ss", "ẞ": "SS", "đ": "d", "Đ": "D", "ð": "d", "Ð": "D", "þ": "th", "Þ": "th", "ł": "l", "Ł": "L", } def remove_symbols_and_diacritics(s: str, keep=""): """ Replace any other markers, symbols, and punctuations with a space, and drop any diacritics (category 'Mn' and some manual mappings) """ return "".join( ( c if c in keep else ( ADDITIONAL_DIACRITICS[c] if c in ADDITIONAL_DIACRITICS else ( "" if unicodedata.category(c) == "Mn" else " " if unicodedata.category(c)[0] in "MSP" else c ) ) ) for c in unicodedata.normalize("NFKD", s) ) def remove_symbols(s: str): """ Replace any other markers, symbols, punctuations with a space, keeping diacritics """ return "".join( " " if unicodedata.category(c)[0] in "MSP" else c for c in unicodedata.normalize("NFKC", s) ) class BasicTextNormalizer: def __init__(self, remove_diacritics: bool = False, split_letters: bool = False): self.clean = ( remove_symbols_and_diacritics if remove_diacritics else remove_symbols ) self.split_letters = split_letters def __call__(self, s: str): s = s.lower() s = re.sub(r"[<\[][^>\]]*[>\]]", "", s) # remove words between brackets s = re.sub(r"\(([^)]+?)\)", "", s) # remove words between parenthesis s = self.clean(s).lower() if self.split_letters: s = " ".join(regex.findall(r"\X", s, regex.U)) s = re.sub( r"\s+", " ", s ) # replace any successive whitespace characters with a space return s - whisper/normalizers/english.json (56128 bytes)
{ "accessorise": "accessorize", "accessorised": "accessorized", "accessorises": "accessorizes", "accessorising": "accessorizing", "acclimatisation": "acclimatization", "acclimatise": "acclimatize", "acclimatised": "acclimatized", "acclimatises": "acclimatizes", "acclimatising": "acclimatizing", "accoutrements": "accouterments", "aeon": "eon", "aeons": "eons", "aerogramme": "aerogram", "aerogrammes": "aerograms", "aeroplane": "airplane", "aeroplanes": "airplanes", "aesthete": "esthete", "aesthetes": "esthetes", "aesthetic": "esthetic", "aesthetically": "esthetically", "aesthetics": "esthetics", "aetiology": "etiology", "ageing": "aging", "aggrandisement": "aggrandizement", "agonise": "agonize", "agonised": "agonized", "agonises": "agonizes", "agonising": "agonizing", "agonisingly": "agonizingly", "almanack": "almanac", "almanacks": "almanacs", "aluminium": "aluminum", "amortisable": "amortizable", "amortisation": "amortization", "amortisations": "amortizations", "amortise": "amortize", "amortised": "amortized", "amortises": "amortizes", "amortising": "amortizing", "amphitheatre": "amphitheater", "amphitheatres": "amphitheaters", "anaemia": "anemia", "anaemic": "anemic", "anaesthesia": "anesthesia", "anaesthetic": "anesthetic", "anaesthetics": "anesthetics", "anaesthetise": "anesthetize", "anaesthetised": "anesthetized", "anaesthetises": "anesthetizes", "anaesthetising": "anesthetizing", "anaesthetist": "anesthetist", "anaesthetists": "anesthetists", "anaesthetize": "anesthetize", "anaesthetized": "anesthetized", "anaesthetizes": "anesthetizes", "anaesthetizing": "anesthetizing", "analogue": "analog", "analogues": "analogs", "analyse": "analyze", "analysed": "analyzed", "analyses": "analyzes", "analysing": "analyzing", "anglicise": "anglicize", "anglicised": "anglicized", "anglicises": "anglicizes", "anglicising": "anglicizing", "annualised": "annualized", "antagonise": "antagonize", "antagonised": "antagonized", "antagonises": "antagonizes", "antagonising": "antagonizing", "apologise": "apologize", "apologised": "apologized", "apologises": "apologizes", "apologising": "apologizing", "appal": "appall", "appals": "appalls", "appetiser": "appetizer", "appetisers": "appetizers", "appetising": "appetizing", "appetisingly": "appetizingly", "arbour": "arbor", "arbours": "arbors", "archeological": "archaeological", "archaeologically": "archeologically", "archaeologist": "archeologist", "archaeologists": "archeologists", "archaeology": "archeology</span>", "ardour": "ardor", "armour": "armor", "armoured": "armored", "armourer": "armorer", "armourers": "armorers", "armouries": "armories", "armoury": "armory", "artefact": "artifact", "artefacts": "artifacts", "authorise": "authorize", "authorised": "authorized", "authorises": "authorizes", "authorising": "authorizing", "axe": "ax", "backpedalled": "backpedaled", "backpedalling": "backpedaling", "bannister": "banister", "bannisters": "banisters", "baptise": "baptize", "baptised": "baptized", "baptises": "baptizes", "baptising": "baptizing", "bastardise": "bastardize", "bastardised": "bastardized", "bastardises": "bastardizes", "bastardising": "bastardizing", "battleax": "battleaxe", "baulk": "balk", "baulked": "balked", "baulking": "balking", "baulks": "balks", "bedevilled": "bedeviled", "bedevilling": "bedeviling", "behaviour": "behavior", "behavioural": "behavioral", "behaviourism": "behaviorism", "behaviourist": "behaviorist", "behaviourists": "behaviorists", "behaviours": "behaviors", "behove": "behoove", "behoved": "behooved", "behoves": "behooves", "bejewelled": "bejeweled", "belabour": "belabor", "belaboured": "belabored", "belabouring": "belaboring", "belabours": "belabors", "bevelled": "beveled", "bevvies": "bevies", "bevvy": "bevy", "biassed": "biased", "biassing": "biasing", "bingeing": "binging", "bougainvillaea": "bougainvillea", "bougainvillaeas": "bougainvilleas", "bowdlerise": "bowdlerize", "bowdlerised": "bowdlerized", "bowdlerises": "bowdlerizes", "bowdlerising": "bowdlerizing", "breathalyse": "breathalyze", "breathalysed": "breathalyzed", "breathalyser": "breathalyzer", "breathalysers": "breathalyzers", "breathalyses": "breathalyzes", "breathalysing": "breathalyzing", "brutalise": "brutalize", "brutalised": "brutalized", "brutalises": "brutalizes", "brutalising": "brutalizing", "busses": "buses", "bussing": "busing", "caesarean": "cesarean", "caesareans": "cesareans", "calibre": "caliber", "calibres": "calibers", "calliper": "caliper", "callipers": "calipers", "callisthenics": "calisthenics", "canalise": "canalize", "canalised": "canalized", "canalises": "canalizes", "canalising": "canalizing", "cancelation": "cancellation", "cancelations": "cancellations", "cancelled": "canceled", "cancelling": "canceling", "candour": "candor", "cannibalise": "cannibalize", "cannibalised": "cannibalized", "cannibalises": "cannibalizes", "cannibalising": "cannibalizing", "canonise": "canonize", "canonised": "canonized", "canonises": "canonizes", "canonising": "canonizing", "capitalise": "capitalize", "capitalised": "capitalized", "capitalises": "capitalizes", "capitalising": "capitalizing", "caramelise": "caramelize", "caramelised": "caramelized", "caramelises": "caramelizes", "caramelising": "caramelizing", "carbonise": "carbonize", "carbonised": "carbonized", "carbonises": "carbonizes", "carbonising": "carbonizing", "carolled": "caroled", "carolling": "caroling", "catalogue": "catalog", "catalogued": "cataloged", "catalogues": "catalogs", "cataloguing": "cataloging", "catalyse": "catalyze", "catalysed": "catalyzed", "catalyses": "catalyzes", "catalysing": "catalyzing", "categorise": "categorize", "categorised": "categorized", "categorises": "categorizes", "categorising": "categorizing", "cauterise": "cauterize", "cauterised": "cauterized", "cauterises": "cauterizes", "cauterising": "cauterizing", "cavilled": "caviled", "cavilling": "caviling", "centigramme": "centigram", "centigrammes": "centigrams", "centilitre": "centiliter", "centilitres": "centiliters", "centimetre": "centimeter", "centimetres": "centimeters", "centralise": "centralize", "centralised": "centralized", "centralises": "centralizes", "centralising": "centralizing", "centre": "center", "centred": "centered", "centrefold": "centerfold", "centrefolds": "centerfolds", "centrepiece": "centerpiece", "centrepieces": "centerpieces", "centres": "centers", "channelled": "channeled", "channelling": "channeling", "characterise": "characterize", "characterised": "characterized", "characterises": "characterizes", "characterising": "characterizing", "cheque": "check", "chequebook": "checkbook", "chequebooks": "checkbooks", "chequered": "checkered", "cheques": "checks", "chilli": "chili", "chimaera": "chimera", "chimaeras": "chimeras", "chiselled": "chiseled", "chiselling": "chiseling", "circularise": "circularize", "circularised": "circularized", "circularises": "circularizes", "circularising": "circularizing", "civilise": "civilize", "civilised": "civilized", "civilises": "civilizes", "civilising": "civilizing", "clamour": "clamor", "clamoured": "clamored", "clamouring": "clamoring", "clamours": "clamors", "clangour": "clangor", "clarinettist": "clarinetist", "clarinettists": "clarinetists", "collectivise": "collectivize", "collectivised": "collectivized", "collectivises": "collectivizes", "collectivising": "collectivizing", "colonisation": "colonization", "colonise": "colonize", "colonised": "colonized", "coloniser": "colonizer", "colonisers": "colonizers", "colonises": "colonizes", "colonising": "colonizing", "colour": "color", "colourant": "colorant", "colourants": "colorants", "coloured": "colored", "coloureds": "coloreds", "colourful": "colorful", "colourfully": "colorfully", "colouring": "coloring", "colourize": "colorize", "colourized": "colorized", "colourizes": "colorizes", "colourizing": "colorizing", "colourless": "colorless", "colours": "colors", "commercialise": "commercialize", "commercialised": "commercialized", "commercialises": "commercializes", "commercialising": "commercializing", "compartmentalise": "compartmentalize", "compartmentalised": "compartmentalized", "compartmentalises": "compartmentalizes", "compartmentalising": "compartmentalizing", "computerise": "computerize", "computerised": "computerized", "computerises": "computerizes", "computerising": "computerizing", "conceptualise": "conceptualize", "conceptualised": "conceptualized", "conceptualises": "conceptualizes", "conceptualising": "conceptualizing", "connexion": "connection", "connexions": "connections", "contextualise": "contextualize", "contextualised": "contextualized", "contextualises": "contextualizes", "contextualising": "contextualizing", "cosier": "cozier", "cosies": "cozies", "cosiest": "coziest", "cosily": "cozily", "cosiness": "coziness", "cosy": "cozy", "councillor": "councilor", "councillors": "councilors", "counselled": "counseled", "counselling": "counseling", "counsellor": "counselor", "counsellors": "counselors", "crenelated": "crenellated", "criminalise": "criminalize", "criminalised": "criminalized", "criminalises": "criminalizes", "criminalising": "criminalizing", "criticise": "criticize", "criticised": "criticized", "criticises": "criticizes", "criticising": "criticizing", "crueller": "crueler", "cruellest": "cruelest", "crystallisation": "crystallization", "crystallise": "crystallize", "crystallised": "crystallized", "crystallises": "crystallizes", "crystallising": "crystallizing", "cudgelled": "cudgeled", "cudgelling": "cudgeling", "customise": "customize", "customised": "customized", "customises": "customizes", "customising": "customizing", "cypher": "cipher", "cyphers": "ciphers", "decentralisation": "decentralization", "decentralise": "decentralize", "decentralised": "decentralized", "decentralises": "decentralizes", "decentralising": "decentralizing", "decriminalisation": "decriminalization", "decriminalise": "decriminalize", "decriminalised": "decriminalized", "decriminalises": "decriminalizes", "decriminalising": "decriminalizing", "defence": "defense", "defenceless": "defenseless", "defences": "defenses", "dehumanisation": "dehumanization", "dehumanise": "dehumanize", "dehumanised": "dehumanized", "dehumanises": "dehumanizes", "dehumanising": "dehumanizing", "demeanour": "demeanor", "demilitarisation": "demilitarization", "demilitarise": "demilitarize", "demilitarised": "demilitarized", "demilitarises": "demilitarizes", "demilitarising": "demilitarizing", "demobilisation": "demobilization", "demobilise": "demobilize", "demobilised": "demobilized", "demobilises": "demobilizes", "demobilising": "demobilizing", "democratisation": "democratization", "democratise": "democratize", "democratised": "democratized", "democratises": "democratizes", "democratising": "democratizing", "demonise": "demonize", "demonised": "demonized", "demonises": "demonizes", "demonising": "demonizing", "demoralisation": "demoralization", "demoralise": "demoralize", "demoralised": "demoralized", "demoralises": "demoralizes", "demoralising": "demoralizing", "denationalisation": "denationalization", "denationalise": "denationalize", "denationalised": "denationalized", "denationalises": "denationalizes", "denationalising": "denationalizing", "deodorise": "deodorize", "deodorised": "deodorized", "deodorises": "deodorizes", "deodorising": "deodorizing", "depersonalise": "depersonalize", "depersonalised": "depersonalized", "depersonalises": "depersonalizes", "depersonalising": "depersonalizing", "deputise": "deputize", "deputised": "deputized", "deputises": "deputizes", "deputising": "deputizing", "desensitisation": "desensitization", "desensitise": "desensitize", "desensitised": "desensitized", "desensitises": "desensitizes", "desensitising": "desensitizing", "destabilisation": "destabilization", "destabilise": "destabilize", "destabilised": "destabilized", "destabilises": "destabilizes", "destabilising": "destabilizing", "dialled": "dialed", "dialling": "dialing", "dialogue": "dialog", "dialogues": "dialogs", "diarrhoea": "diarrhea", "digitise": "digitize", "digitised": "digitized", "digitises": "digitizes", "digitising": "digitizing", "disc": "disk", "discolour": "discolor", "discoloured": "discolored", "discolouring": "discoloring", "discolours": "discolors", "discs": "disks", "disembowelled": "disemboweled", "disembowelling": "disemboweling", "disfavour": "disfavor", "dishevelled": "disheveled", "dishonour": "dishonor", "dishonourable": "dishonorable", "dishonourably": "dishonorably", "dishonoured": "dishonored", "dishonouring": "dishonoring", "dishonours": "dishonors", "disorganisation": "disorganization", "disorganised": "disorganized", "distil": "distill", "distils": "distills", "dramatisation": "dramatization", "dramatisations": "dramatizations", "dramatise": "dramatize", "dramatised": "dramatized", "dramatises": "dramatizes", "dramatising": "dramatizing", "draught": "draft", "draughtboard": "draftboard", "draughtboards": "draftboards", "draughtier": "draftier", "draughtiest": "draftiest", "draughts": "drafts", "draughtsman": "draftsman", "draughtsmanship": "draftsmanship", "draughtsmen": "draftsmen", "draughtswoman": "drafts - whisper/normalizers/english.py (21001 bytes)
import json import os import re from fractions import Fraction from typing import Iterator, List, Match, Optional, Union from more_itertools import windowed from .basic import remove_symbols_and_diacritics class EnglishNumberNormalizer: """ Convert any spelled-out numbers into arabic numbers, while handling: - remove any commas - keep the suffixes such as: `1960s`, `274th`, `32nd`, etc. - spell out currency symbols after the number. e.g. `$20 million` -> `20000000 dollars` - spell out `one` and `ones` - interpret successive single-digit numbers as nominal: `one oh one` -> `101` """ def __init__(self): super().__init__() self.zeros = {"o", "oh", "zero"} self.ones = { name: i for i, name in enumerate( [ "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten", "eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen", "seventeen", "eighteen", "nineteen", ], start=1, ) } self.ones_plural = { "sixes" if name == "six" else name + "s": (value, "s") for name, value in self.ones.items() } self.ones_ordinal = { "zeroth": (0, "th"), "first": (1, "st"), "second": (2, "nd"), "third": (3, "rd"), "fifth": (5, "th"), "twelfth": (12, "th"), **{ name + ("h" if name.endswith("t") else "th"): (value, "th") for name, value in self.ones.items() if value > 3 and value != 5 and value != 12 }, } self.ones_suffixed = {**self.ones_plural, **self.ones_ordinal} self.tens = { "twenty": 20, "thirty": 30, "forty": 40, "fifty": 50, "sixty": 60, "seventy": 70, "eighty": 80, "ninety": 90, } self.tens_plural = { name.replace("y", "ies"): (value, "s") for name, value in self.tens.items() } self.tens_ordinal = { name.replace("y", "ieth"): (value, "th") for name, value in self.tens.items() } self.tens_suffixed = {**self.tens_plural, **self.tens_ordinal} self.multipliers = { "hundred": 100, "thousand": 1_000, "million": 1_000_000, "billion": 1_000_000_000, "trillion": 1_000_000_000_000, "quadrillion": 1_000_000_000_000_000, "quintillion": 1_000_000_000_000_000_000, "sextillion": 1_000_000_000_000_000_000_000, "septillion": 1_000_000_000_000_000_000_000_000, "octillion": 1_000_000_000_000_000_000_000_000_000, "nonillion": 1_000_000_000_000_000_000_000_000_000_000, "decillion": 1_000_000_000_000_000_000_000_000_000_000_000, } self.multipliers_plural = { name + "s": (value, "s") for name, value in self.multipliers.items() } self.multipliers_ordinal = { name + "th": (value, "th") for name, value in self.multipliers.items() } self.multipliers_suffixed = { **self.multipliers_plural, **self.multipliers_ordinal, } self.decimals = {*self.ones, *self.tens, *self.zeros} self.preceding_prefixers = { "minus": "-", "negative": "-", "plus": "+", "positive": "+", } self.following_prefixers = { "pound": "£", "pounds": "£", "euro": "€", "euros": "€", "dollar": "$", "dollars": "$", "cent": "¢", "cents": "¢", } self.prefixes = set( list(self.preceding_prefixers.values()) + list(self.following_prefixers.values()) ) self.suffixers = { "per": {"cent": "%"}, "percent": "%", } self.specials = {"and", "double", "triple", "point"} self.words = set( [ key for mapping in [ self.zeros, self.ones, self.ones_suffixed, self.tens, self.tens_suffixed, self.multipliers, self.multipliers_suffixed, self.preceding_prefixers, self.following_prefixers, self.suffixers, self.specials, ] for key in mapping ] ) self.literal_words = {"one", "ones"} def process_words(self, words: List[str]) -> Iterator[str]: prefix: Optional[str] = None value: Optional[Union[str, int]] = None skip = False def to_fraction(s: str): try: return Fraction(s) except ValueError: return None def output(result: Union[str, int]): nonlocal prefix, value result = str(result) if prefix is not None: result = prefix + result value = None prefix = None return result if len(words) == 0: return for prev, current, next in windowed([None] + words + [None], 3): if skip: skip = False continue next_is_numeric = next is not None and re.match(r"^\d+(\.\d+)?$", next) has_prefix = current[0] in self.prefixes current_without_prefix = current[1:] if has_prefix else current if re.match(r"^\d+(\.\d+)?$", current_without_prefix): # arabic numbers (potentially with signs and fractions) f = to_fraction(current_without_prefix) assert f is not None if value is not None: if isinstance(value, str) and value.endswith("."): # concatenate decimals / ip address components value = str(value) + str(current) continue else: yield output(value) prefix = current[0] if has_prefix else prefix if f.denominator == 1: value = f.numerator # store integers as int else: value = current_without_prefix elif current not in self.words: # non-numeric words if value is not None: yield output(value) yield output(current) elif current in self.zeros: value = str(value or "") + "0" elif current in self.ones: ones = self.ones[current] if value is None: value = ones elif isinstance(value, str) or prev in self.ones: if ( prev in self.tens and ones < 10 ): # replace the last zero with the digit assert value[-1] == "0" value = value[:-1] + str(ones) else: value = str(value) + str(ones) elif ones < 10: if value % 10 == 0: value += ones else: value = str(value) + str(ones) else: # eleven to nineteen if value % 100 == 0: value += ones else: value = str(value) + str(ones) elif current in self.ones_suffixed: # ordinal or cardinal; yield the number right away ones, suffix = self.ones_suffixed[current] if value is None: yield output(str(ones) + suffix) elif isinstance(value, str) or prev in self.ones: if prev in self.tens and ones < 10: assert value[-1] == "0" yield output(value[:-1] + str(ones) + suffix) else: yield output(str(value) + str(ones) + suffix) elif ones < 10: if value % 10 == 0: yield output(str(value + ones) + suffix) else: yield output(str(value) + str(ones) + suffix) else: # eleven to nineteen if value % 100 == 0: yield output(str(value + ones) + suffix) else: yield output(str(value) + str(ones) + suffix) value = None elif current in self.tens: tens = self.tens[current] if value is None: value = tens elif isinstance(value, str): value = str(value) + str(tens) else: if value % 100 == 0: value += tens else: value = str(value) + str(tens) elif current in self.tens_suffixed: # ordinal or cardinal; yield the number right away tens, suffix = self.tens_suffixed[current] if value is None: yield output(str(tens) + suffix) elif isinstance(value, str): yield output(str(value) + str(tens) + suffix) else: if value % 100 == 0: yield output(str(value + tens) + suffix) else: yield output(str(value) + str(tens) + suffix) elif current in self.multipliers: multiplier = self.multipliers[current] if value is None: value = multiplier elif isinstance(value, str) or value == 0: f = to_fraction(value) p = f * multiplier if f is not None else None if f is not None and p.denominator == 1: value = p.numerator else: yield output(value) value = multiplier else: before = value // 1000 * 1000 residual = value % 1000 value = before + residual * multiplier elif current in self.multipliers_suffixed: multiplier, suffix = self.multipliers_suffixed[current] if value is None: yield output(str(multiplier) + suffix) elif isinstance(value, str): f = to_fraction(value) p = f * multiplier if f is not None else None if f is not None and p.denominator == 1: yield output(str(p.numerator) + suffix) else: yield output(value) yield output(str(multiplier) + suffix) else: # int before = value // 1000 * 1000 residual = value % 1000 value = before + residual * multiplier yield output(str(value) + suffix) value = None elif current in self.preceding_prefixers: # apply prefix (positive, minus, etc.) if it precedes a number if value is not None: yield output(value) if next in self.words or next_is_numeric: prefix = self.preceding_prefixers[current] else: yield output(current) elif current in self.following_prefixers: # apply prefix (dollars, cents, etc.) only after a number if value is not None: prefix = self.following_prefixers[current] yield output(value) else: yield output(current) elif current in self.suffixers: # apply suffix symbols (percent -> '%') if value is not None: suffix = self.suffixers[current] if isinstance(suffix, dict): if next in suffix: yield output(str(value) + suffix[next]) skip = True else: yield output(value) yield output(current) else: yield output(str(value) + suffix) else: yield output(current) elif current in self.specials: if next not in self.words and not next_is_numeric: # apply special handling only if the next word can be numeric if value is not None: yield output(value) yield output(current) elif current == "and": # ignore "and" after hundreds, thousands, etc. if prev not in self.multipliers: if value is not None: yield output(value) yield output(current) elif current == "double" or current == "triple": if next in self.ones or next in self.zeros: repeats = 2 if current == "double" else 3 ones = self.ones.get(next, 0) value = str(value or "") + str(ones) * repeats skip = True else: if value is not None: yield output(value) yield output(current) elif current == "point": if next in self.decimals or next_is_numeric: value = str(value or "") + "." else: # should all have been covered at this point raise ValueError(f"Unexpected token: {current}") else: # all should have been covered at this point raise ValueError(f"Unexpected token: {current}") if value is not None: yield output(value - whisper/timing.py (12697 bytes)
import itertools import subprocess import warnings from dataclasses import dataclass from typing import TYPE_CHECKING, List import numba import numpy as np import torch import torch.nn.functional as F from .audio import HOP_LENGTH, SAMPLE_RATE, TOKENS_PER_SECOND from .tokenizer import Tokenizer if TYPE_CHECKING: from .model import Whisper def median_filter(x: torch.Tensor, filter_width: int): """Apply a median filter of width `filter_width` along the last dimension of `x`""" pad_width = filter_width // 2 if x.shape[-1] <= pad_width: # F.pad requires the padding width to be smaller than the input dimension return x if (ndim := x.ndim) <= 2: # `F.pad` does not support 1D or 2D inputs for reflect padding but supports 3D and 4D x = x[None, None, :] assert ( filter_width > 0 and filter_width % 2 == 1 ), "`filter_width` should be an odd number" result = None x = F.pad(x, (filter_width // 2, filter_width // 2, 0, 0), mode="reflect") if x.is_cuda: try: from .triton_ops import median_filter_cuda result = median_filter_cuda(x, filter_width) except (RuntimeError, subprocess.CalledProcessError): warnings.warn( "Failed to launch Triton kernels, likely due to missing CUDA toolkit; " "falling back to a slower median kernel implementation..." ) if result is None: # sort() is faster than torch.median (https://github.com/pytorch/pytorch/issues/51450) result = x.unfold(-1, filter_width, 1).sort()[0][..., filter_width // 2] if ndim <= 2: result = result[0, 0] return result @numba.jit(nopython=True) def backtrace(trace: np.ndarray): i = trace.shape[0] - 1 j = trace.shape[1] - 1 trace[0, :] = 2 trace[:, 0] = 1 result = [] while i > 0 or j > 0: result.append((i - 1, j - 1)) if trace[i, j] == 0: i -= 1 j -= 1 elif trace[i, j] == 1: i -= 1 elif trace[i, j] == 2: j -= 1 else: raise ValueError("Unexpected trace[i, j]") result = np.array(result) return result[::-1, :].T @numba.jit(nopython=True, parallel=True) def dtw_cpu(x: np.ndarray): N, M = x.shape cost = np.ones((N + 1, M + 1), dtype=np.float32) * np.inf trace = -np.ones((N + 1, M + 1), dtype=np.float32) cost[0, 0] = 0 for j in range(1, M + 1): for i in range(1, N + 1): c0 = cost[i - 1, j - 1] c1 = cost[i - 1, j] c2 = cost[i, j - 1] if c0 < c1 and c0 < c2: c, t = c0, 0 elif c1 < c0 and c1 < c2: c, t = c1, 1 else: c, t = c2, 2 cost[i, j] = x[i - 1, j - 1] + c trace[i, j] = t return backtrace(trace) def dtw_cuda(x, BLOCK_SIZE=1024): from .triton_ops import dtw_kernel M, N = x.shape assert M < BLOCK_SIZE, f"M should be smaller than {BLOCK_SIZE=}" x_skew = ( F.pad(x, (0, M + 1), value=np.inf).flatten()[: M * (N + M)].reshape(M, N + M) ) x_skew = x_skew.T.contiguous() cost = torch.ones(N + M + 2, M + 2) * np.inf cost[0, 0] = 0 cost = cost.to(x.device) trace = torch.zeros_like(cost, dtype=torch.int32) dtw_kernel[(1,)]( cost, trace, x_skew, x_skew.stride(0), cost.stride(0), trace.stride(0), N, M, BLOCK_SIZE=BLOCK_SIZE, ) trace = trace.T.flatten()[: (M + 1) * (M + N + 3)].reshape(M + 1, M + N + 3)[ :, : N + 1 ] return backtrace(trace.cpu().numpy()) def dtw(x: torch.Tensor) -> np.ndarray: if x.is_cuda: try: return dtw_cuda(x) except (RuntimeError, subprocess.CalledProcessError): warnings.warn( "Failed to launch Triton kernels, likely due to missing CUDA toolkit; " "falling back to a slower DTW implementation..." ) return dtw_cpu(x.double().cpu().numpy()) @dataclass class WordTiming: word: str tokens: List[int] start: float end: float probability: float def find_alignment( model: "Whisper", tokenizer: Tokenizer, text_tokens: List[int], mel: torch.Tensor, num_frames: int, *, medfilt_width: int = 7, qk_scale: float = 1.0, ) -> List[WordTiming]: if len(text_tokens) == 0: return [] tokens = torch.tensor( [ *tokenizer.sot_sequence, tokenizer.no_timestamps, *text_tokens, tokenizer.eot, ] ).to(model.device) # install hooks on the cross attention layers to retrieve the attention weights QKs = [None] * model.dims.n_text_layer hooks = [ block.cross_attn.register_forward_hook( lambda _, ins, outs, index=i: QKs.__setitem__(index, outs[-1][0]) ) for i, block in enumerate(model.decoder.blocks) ] from .model import disable_sdpa with torch.no_grad(), disable_sdpa(): logits = model(mel.unsqueeze(0), tokens.unsqueeze(0))[0] sampled_logits = logits[len(tokenizer.sot_sequence) :, : tokenizer.eot] token_probs = sampled_logits.softmax(dim=-1) text_token_probs = token_probs[np.arange(len(text_tokens)), text_tokens] text_token_probs = text_token_probs.tolist() for hook in hooks: hook.remove() # heads * tokens * frames weights = torch.stack([QKs[_l][_h] for _l, _h in model.alignment_heads.indices().T]) weights = weights[:, :, : num_frames // 2] weights = (weights * qk_scale).softmax(dim=-1) std, mean = torch.std_mean(weights, dim=-2, keepdim=True, unbiased=False) weights = (weights - mean) / std weights = median_filter(weights, medfilt_width) matrix = weights.mean(axis=0) matrix = matrix[len(tokenizer.sot_sequence) : -1] text_indices, time_indices = dtw(-matrix) words, word_tokens = tokenizer.split_to_word_tokens(text_tokens + [tokenizer.eot]) if len(word_tokens) <= 1: # return on eot only # >>> np.pad([], (1, 0)) # array([0.]) # This results in crashes when we lookup jump_times with float, like # IndexError: arrays used as indices must be of integer (or boolean) type return [] word_boundaries = np.pad(np.cumsum([len(t) for t in word_tokens[:-1]]), (1, 0)) jumps = np.pad(np.diff(text_indices), (1, 0), constant_values=1).astype(bool) jump_times = time_indices[jumps] / TOKENS_PER_SECOND start_times = jump_times[word_boundaries[:-1]] end_times = jump_times[word_boundaries[1:]] word_probabilities = [ np.mean(text_token_probs[i:j]) for i, j in zip(word_boundaries[:-1], word_boundaries[1:]) ] return [ WordTiming(word, tokens, start, end, probability) for word, tokens, start, end, probability in zip( words, word_tokens, start_times, end_times, word_probabilities ) ] def merge_punctuations(alignment: List[WordTiming], prepended: str, appended: str): # merge prepended punctuations i = len(alignment) - 2 j = len(alignment) - 1 while i >= 0: previous = alignment[i] following = alignment[j] if previous.word.startswith(" ") and previous.word.strip() in prepended: # prepend it to the following word following.word = previous.word + following.word following.tokens = previous.tokens + following.tokens previous.word = "" previous.tokens = [] else: j = i i -= 1 # merge appended punctuations i = 0 j = 1 while j < len(alignment): previous = alignment[i] following = alignment[j] if not previous.word.endswith(" ") and following.word in appended: # append it to the previous word previous.word = previous.word + following.word previous.tokens = previous.tokens + following.tokens following.word = "" following.tokens = [] else: i = j j += 1 def add_word_timestamps( *, segments: List[dict], model: "Whisper", tokenizer: Tokenizer, mel: torch.Tensor, num_frames: int, prepend_punctuations: str = "\"'“¿([{-", append_punctuations: str = "\"'.。,,!!??::”)]}、", last_speech_timestamp: float, **kwargs, ): if len(segments) == 0: return text_tokens_per_segment = [ [token for token in segment["tokens"] if token < tokenizer.eot] for segment in segments ] text_tokens = list(itertools.chain.from_iterable(text_tokens_per_segment)) alignment = find_alignment(model, tokenizer, text_tokens, mel, num_frames, **kwargs) word_durations = np.array([t.end - t.start for t in alignment]) word_durations = word_durations[word_durations.nonzero()] median_duration = np.median(word_durations) if len(word_durations) > 0 else 0.0 median_duration = min(0.7, float(median_duration)) max_duration = median_duration * 2 # hack: truncate long words at sentence boundaries. # a better segmentation algorithm based on VAD should be able to replace this. if len(word_durations) > 0: sentence_end_marks = ".。!!??" # ensure words at sentence boundaries are not longer than twice the median word duration. for i in range(1, len(alignment)): if alignment[i].end - alignment[i].start > max_duration: if alignment[i].word in sentence_end_marks: alignment[i].end = alignment[i].start + max_duration elif alignment[i - 1].word in sentence_end_marks: alignment[i].start = alignment[i].end - max_duration merge_punctuations(alignment, prepend_punctuations, append_punctuations) time_offset = segments[0]["seek"] * HOP_LENGTH / SAMPLE_RATE word_index = 0 for segment, text_tokens in zip(segments, text_tokens_per_segment): saved_tokens = 0 words = [] while word_index < len(alignment) and saved_tokens < len(text_tokens): timing = alignment[word_index] if timing.word: words.append( dict( word=timing.word, start=round(time_offset + timing.start, 2), end=round(time_offset + timing.end, 2), probability=timing.probability, ) ) saved_tokens += len(timing.tokens) word_index += 1 # hack: truncate long words at segment boundaries. # a better segmentation algorithm based on VAD should be able to replace this. if len(words) > 0: # ensure the first and second word after a pause is not longer than # twice the median word duration. if words[0]["end"] - last_speech_timestamp > median_duration * 4 and ( words[0]["end"] - words[0]["start"] > max_duration or ( len(words) > 1 and words[1]["end"] - words[0]["start"] > max_duration * 2 ) ): if ( len(words) > 1 and words[1]["end"] - words[1]["start"] > max_duration ): boundary = max(words[1]["end"] / 2, words[1]["end"] - max_duration) words[0]["end"] = words[1]["start"] = boundary words[0]["start"] = max(0, words[0]["end"] - max_duration) # prefer the segment-level start timestamp if the first word is too long. if ( segment["start"] < words[0]["end"] and segment["start"] - 0.5 > words[0]["start"] ): words[0]["start"] = max( 0, min(words[0]["end"] - median_duration, segment["start"]) ) else: segment["start"] = words[0]["start"] # prefer the segment-level end timestamp if the last word is too long. if ( segment["end"] > words[-1]["start"] and segment["end"] + 0.5 < words[-1]["end"] ): words[-1]["end"] = max( words[-1]["start"] + median_duration, segment["end"] ) else: segment["end"] = words[-1]["end"] last_speech_timestamp = segment["end"] segment["words"] = words - whisper/tokenizer.py (12338 bytes)
import base64 import os import string from dataclasses import dataclass, field from functools import cached_property, lru_cache from typing import Dict, List, Optional, Tuple import tiktoken LANGUAGES = { "en": "english", "zh": "chinese", "de": "german", "es": "spanish", "ru": "russian", "ko": "korean", "fr": "french", "ja": "japanese", "pt": "portuguese", "tr": "turkish", "pl": "polish", "ca": "catalan", "nl": "dutch", "ar": "arabic", "sv": "swedish", "it": "italian", "id": "indonesian", "hi": "hindi", "fi": "finnish", "vi": "vietnamese", "he": "hebrew", "uk": "ukrainian", "el": "greek", "ms": "malay", "cs": "czech", "ro": "romanian", "da": "danish", "hu": "hungarian", "ta": "tamil", "no": "norwegian", "th": "thai", "ur": "urdu", "hr": "croatian", "bg": "bulgarian", "lt": "lithuanian", "la": "latin", "mi": "maori", "ml": "malayalam", "cy": "welsh", "sk": "slovak", "te": "telugu", "fa": "persian", "lv": "latvian", "bn": "bengali", "sr": "serbian", "az": "azerbaijani", "sl": "slovenian", "kn": "kannada", "et": "estonian", "mk": "macedonian", "br": "breton", "eu": "basque", "is": "icelandic", "hy": "armenian", "ne": "nepali", "mn": "mongolian", "bs": "bosnian", "kk": "kazakh", "sq": "albanian", "sw": "swahili", "gl": "galician", "mr": "marathi", "pa": "punjabi", "si": "sinhala", "km": "khmer", "sn": "shona", "yo": "yoruba", "so": "somali", "af": "afrikaans", "oc": "occitan", "ka": "georgian", "be": "belarusian", "tg": "tajik", "sd": "sindhi", "gu": "gujarati", "am": "amharic", "yi": "yiddish", "lo": "lao", "uz": "uzbek", "fo": "faroese", "ht": "haitian creole", "ps": "pashto", "tk": "turkmen", "nn": "nynorsk", "mt": "maltese", "sa": "sanskrit", "lb": "luxembourgish", "my": "myanmar", "bo": "tibetan", "tl": "tagalog", "mg": "malagasy", "as": "assamese", "tt": "tatar", "haw": "hawaiian", "ln": "lingala", "ha": "hausa", "ba": "bashkir", "jw": "javanese", "su": "sundanese", "yue": "cantonese", } # language code lookup by name, with a few language aliases TO_LANGUAGE_CODE = { **{language: code for code, language in LANGUAGES.items()}, "burmese": "my", "valencian": "ca", "flemish": "nl", "haitian": "ht", "letzeburgesch": "lb", "pushto": "ps", "panjabi": "pa", "moldavian": "ro", "moldovan": "ro", "sinhalese": "si", "castilian": "es", "mandarin": "zh", } @dataclass class Tokenizer: """A thin wrapper around `tiktoken` providing quick access to special tokens""" encoding: tiktoken.Encoding num_languages: int language: Optional[str] = None task: Optional[str] = None sot_sequence: Tuple[int] = () special_tokens: Dict[str, int] = field(default_factory=dict) def __post_init__(self): for special in self.encoding.special_tokens_set: special_token = self.encoding.encode_single_token(special) self.special_tokens[special] = special_token sot: int = self.special_tokens["<|startoftranscript|>"] translate: int = self.special_tokens["<|translate|>"] transcribe: int = self.special_tokens["<|transcribe|>"] langs = tuple(LANGUAGES.keys())[: self.num_languages] sot_sequence = [sot] if self.language is not None: sot_sequence.append(sot + 1 + langs.index(self.language)) if self.task is not None: task_token: int = transcribe if self.task == "transcribe" else translate sot_sequence.append(task_token) self.sot_sequence = tuple(sot_sequence) def encode(self, text, **kwargs): return self.encoding.encode(text, **kwargs) def decode(self, token_ids: List[int], **kwargs) -> str: token_ids = [t for t in token_ids if t < self.timestamp_begin] return self.encoding.decode(token_ids, **kwargs) def decode_with_timestamps(self, token_ids: List[int], **kwargs) -> str: """ Timestamp tokens are above other special tokens' id range and are ignored by `decode()`. This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>". """ return self.encoding.decode(token_ids, **kwargs) @cached_property def eot(self) -> int: return self.encoding.eot_token @cached_property def transcribe(self) -> int: return self.special_tokens["<|transcribe|>"] @cached_property def translate(self) -> int: return self.special_tokens["<|translate|>"] @cached_property def sot(self) -> int: return self.special_tokens["<|startoftranscript|>"] @cached_property def sot_lm(self) -> int: return self.special_tokens["<|startoflm|>"] @cached_property def sot_prev(self) -> int: return self.special_tokens["<|startofprev|>"] @cached_property def no_speech(self) -> int: return self.special_tokens["<|nospeech|>"] @cached_property def no_timestamps(self) -> int: return self.special_tokens["<|notimestamps|>"] @cached_property def timestamp_begin(self) -> int: return self.special_tokens["<|0.00|>"] @cached_property def language_token(self) -> int: """Returns the token id corresponding to the value of the `language` field""" if self.language is None: raise ValueError("This tokenizer does not have language token configured") return self.to_language_token(self.language) def to_language_token(self, language): if token := self.special_tokens.get(f"<|{language}|>", None): return token raise KeyError(f"Language {language} not found in tokenizer.") @cached_property def all_language_tokens(self) -> Tuple[int]: result = [] for token, token_id in self.special_tokens.items(): if token.strip("<|>") in LANGUAGES: result.append(token_id) return tuple(result)[: self.num_languages] @cached_property def all_language_codes(self) -> Tuple[str]: return tuple(self.decode([_l]).strip("<|>") for _l in self.all_language_tokens) @cached_property def sot_sequence_including_notimestamps(self) -> Tuple[int]: return tuple(list(self.sot_sequence) + [self.no_timestamps]) @cached_property def non_speech_tokens(self) -> Tuple[int]: """ Returns the list of tokens to suppress in order to avoid any speaker tags or non-speech annotations, to prevent sampling texts that are not actually spoken in the audio, e.g. - ♪♪♪ - ( SPEAKING FOREIGN LANGUAGE ) - [DAVID] Hey there, keeping basic punctuations like commas, periods, question marks, exclamation points, etc. """ symbols = list('"#()*+/:;<=>@[\\]^_`{|}~「」『』') symbols += ( "<< >> <<< >>> -- --- -( -[ (' (\" (( )) ((( ))) [[ ]] {{ }} ♪♪ ♪♪♪".split() ) # symbols that may be a single token or multiple tokens depending on the tokenizer. # In case they're multiple tokens, suppress the first token, which is safe because: # These are between U+2640 and U+267F miscellaneous symbols that are okay to suppress # in generations, and in the 3-byte UTF-8 representation they share the first two bytes. miscellaneous = set("♩♪♫♬♭♮♯") assert all(0x2640 <= ord(c) <= 0x267F for c in miscellaneous) # allow hyphens "-" and single quotes "'" between words, but not at the beginning of a word result = {self.encoding.encode(" -")[0], self.encoding.encode(" '")[0]} for symbol in symbols + list(miscellaneous): for tokens in [ self.encoding.encode(symbol), self.encoding.encode(" " + symbol), ]: if len(tokens) == 1 or symbol in miscellaneous: result.add(tokens[0]) return tuple(sorted(result)) def split_to_word_tokens(self, tokens: List[int]): if self.language in {"zh", "ja", "th", "lo", "my", "yue"}: # These languages don't typically use spaces, so it is difficult to split words # without morpheme analysis. Here, we instead split words at any # position where the tokens are decoded as valid unicode points return self.split_tokens_on_unicode(tokens) return self.split_tokens_on_spaces(tokens) def split_tokens_on_unicode(self, tokens: List[int]): decoded_full = self.decode_with_timestamps(tokens) replacement_char = "\ufffd" words = [] word_tokens = [] current_tokens = [] unicode_offset = 0 for token in tokens: current_tokens.append(token) decoded = self.decode_with_timestamps(current_tokens) if ( replacement_char not in decoded or decoded_full[unicode_offset + decoded.index(replacement_char)] == replacement_char ): words.append(decoded) word_tokens.append(current_tokens) current_tokens = [] unicode_offset += len(decoded) return words, word_tokens def split_tokens_on_spaces(self, tokens: List[int]): subwords, subword_tokens_list = self.split_tokens_on_unicode(tokens) words = [] word_tokens = [] for subword, subword_tokens in zip(subwords, subword_tokens_list): special = subword_tokens[0] >= self.eot with_space = subword.startswith(" ") punctuation = subword.strip() in string.punctuation if special or with_space or punctuation or len(words) == 0: words.append(subword) word_tokens.append(subword_tokens) else: words[-1] = words[-1] + subword word_tokens[-1].extend(subword_tokens) return words, word_tokens @lru_cache(maxsize=None) def get_encoding(name: str = "gpt2", num_languages: int = 99): vocab_path = os.path.join(os.path.dirname(__file__), "assets", f"{name}.tiktoken") ranks = { base64.b64decode(token): int(rank) for token, rank in (line.split() for line in open(vocab_path) if line) } n_vocab = len(ranks) special_tokens = {} specials = [ "<|endoftext|>", "<|startoftranscript|>", *[f"<|{lang}|>" for lang in list(LANGUAGES.keys())[:num_languages]], "<|translate|>", "<|transcribe|>", "<|startoflm|>", "<|startofprev|>", "<|nospeech|>", "<|notimestamps|>", *[f"<|{i * 0.02:.2f}|>" for i in range(1501)], ] for token in specials: special_tokens[token] = n_vocab n_vocab += 1 return tiktoken.Encoding( name=os.path.basename(vocab_path), explicit_n_vocab=n_vocab, pat_str=r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""", mergeable_ranks=ranks, special_tokens=special_tokens, ) @lru_cache(maxsize=None) def get_tokenizer( multilingual: bool, *, num_languages: int = 99, language: Optional[str] = None, task: Optional[str] = None, # Literal["transcribe", "translate", None] ) -> Tokenizer: if language is not None: language = language.lower() if language not in LANGUAGES: if language in TO_LANGUAGE_CODE: language = TO_LANGUAGE_CODE[language] else: raise ValueError(f"Unsupported language: {language}") if multilingual: encoding_name = "multilingual" language = language or "en" task = task or "transcribe" else: encoding_name = "gpt2" language = None task = None encoding = get_encoding(name=encoding_name, num_languages=num_languages) return Tokenizer( encoding=encoding, num_languages=num_languages, language=language, task=task ) - whisper/transcribe.py (30375 bytes)
import argparse import os import traceback import warnings from typing import TYPE_CHECKING, List, Optional, Tuple, Union import numpy as np import torch import tqdm from .audio import ( FRAMES_PER_SECOND, HOP_LENGTH, N_FRAMES, N_SAMPLES, SAMPLE_RATE, log_mel_spectrogram, pad_or_trim, ) from .decoding import DecodingOptions, DecodingResult from .timing import add_word_timestamps from .tokenizer import LANGUAGES, TO_LANGUAGE_CODE, get_tokenizer from .utils import ( exact_div, format_timestamp, get_end, get_writer, make_safe, optional_float, optional_int, str2bool, ) if TYPE_CHECKING: from .model import Whisper def transcribe( model: "Whisper", audio: Union[str, np.ndarray, torch.Tensor], *, verbose: Optional[bool] = None, temperature: Union[float, Tuple[float, ...]] = (0.0, 0.2, 0.4, 0.6, 0.8, 1.0), compression_ratio_threshold: Optional[float] = 2.4, logprob_threshold: Optional[float] = -1.0, no_speech_threshold: Optional[float] = 0.6, condition_on_previous_text: bool = True, initial_prompt: Optional[str] = None, carry_initial_prompt: bool = False, word_timestamps: bool = False, prepend_punctuations: str = "\"'“¿([{-", append_punctuations: str = "\"'.。,,!!??::”)]}、", clip_timestamps: Union[str, List[float]] = "0", hallucination_silence_threshold: Optional[float] = None, **decode_options, ): """ Transcribe an audio file using Whisper Parameters ---------- model: Whisper The Whisper model instance audio: Union[str, np.ndarray, torch.Tensor] The path to the audio file to open, or the audio waveform verbose: bool Whether to display the text being decoded to the console. If True, displays all the details, If False, displays minimal details. If None, does not display anything temperature: Union[float, Tuple[float, ...]] Temperature for sampling. It can be a tuple of temperatures, which will be successively used upon failures according to either `compression_ratio_threshold` or `logprob_threshold`. compression_ratio_threshold: float If the gzip compression ratio is above this value, treat as failed logprob_threshold: float If the average log probability over sampled tokens is below this value, treat as failed no_speech_threshold: float If the no_speech probability is higher than this value AND the average log probability over sampled tokens is below `logprob_threshold`, consider the segment as silent condition_on_previous_text: bool if True, the previous output of the model is provided as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop, such as repetition looping or timestamps going out of sync. word_timestamps: bool Extract word-level timestamps using the cross-attention pattern and dynamic time warping, and include the timestamps for each word in each segment. prepend_punctuations: str If word_timestamps is True, merge these punctuation symbols with the next word append_punctuations: str If word_timestamps is True, merge these punctuation symbols with the previous word initial_prompt: Optional[str] Optional text to provide as a prompt for the first window. This can be used to provide, or "prompt-engineer" a context for transcription, e.g. custom vocabularies or proper nouns to make it more likely to predict those word correctly. carry_initial_prompt: bool If carry_initial_prompt is True, `initial_prompt` is prepended to the prompt of each internal `decode()` call. If there is not enough context space at the start of the prompt, it is left-sliced to make space. decode_options: dict Keyword arguments to construct `DecodingOptions` instances clip_timestamps: Union[str, List[float]] Comma-separated list start,end,start,end,... timestamps (in seconds) of clips to process. The last end timestamp defaults to the end of the file. hallucination_silence_threshold: Optional[float] When word_timestamps is True, skip silent periods longer than this threshold (in seconds) when a possible hallucination is detected Returns ------- A dictionary containing the resulting text ("text") and segment-level details ("segments"), and the spoken language ("language"), which is detected when `decode_options["language"]` is None. """ dtype = torch.float16 if decode_options.get("fp16", True) else torch.float32 if model.device == torch.device("cpu"): if torch.cuda.is_available(): warnings.warn("Performing inference on CPU when CUDA is available") if dtype == torch.float16: warnings.warn("FP16 is not supported on CPU; using FP32 instead") dtype = torch.float32 if dtype == torch.float32: decode_options["fp16"] = False # Pad 30-seconds of silence to the input audio, for slicing mel = log_mel_spectrogram(audio, model.dims.n_mels, padding=N_SAMPLES) content_frames = mel.shape[-1] - N_FRAMES content_duration = float(content_frames * HOP_LENGTH / SAMPLE_RATE) if decode_options.get("language", None) is None: if not model.is_multilingual: decode_options["language"] = "en" else: if verbose: print( "Detecting language using up to the first 30 seconds. Use `--language` to specify the language" ) mel_segment = pad_or_trim(mel, N_FRAMES).to(model.device).to(dtype) _, probs = model.detect_language(mel_segment) decode_options["language"] = max(probs, key=probs.get) if verbose is not None: print( f"Detected language: {LANGUAGES[decode_options['language']].title()}" ) language: str = decode_options["language"] task: str = decode_options.get("task", "transcribe") tokenizer = get_tokenizer( model.is_multilingual, num_languages=model.num_languages, language=language, task=task, ) if isinstance(clip_timestamps, str): clip_timestamps = [ float(ts) for ts in (clip_timestamps.split(",") if clip_timestamps else []) ] seek_points: List[int] = [round(ts * FRAMES_PER_SECOND) for ts in clip_timestamps] if len(seek_points) == 0: seek_points.append(0) if len(seek_points) % 2 == 1: seek_points.append(content_frames) seek_clips: List[Tuple[int, int]] = list(zip(seek_points[::2], seek_points[1::2])) punctuation = "\"'“¿([{-\"'.。,,!!??::”)]}、" if word_timestamps and task == "translate": warnings.warn("Word-level timestamps on translations may not be reliable.") def decode_with_fallback(segment: torch.Tensor) -> DecodingResult: temperatures = ( [temperature] if isinstance(temperature, (int, float)) else temperature ) decode_result = None for t in temperatures: kwargs = {**decode_options} if t > 0: # disable beam_size and patience when t > 0 kwargs.pop("beam_size", None) kwargs.pop("patience", None) else: # disable best_of when t == 0 kwargs.pop("best_of", None) options = DecodingOptions(**kwargs, temperature=t) decode_result = model.decode(segment, options) needs_fallback = False if ( compression_ratio_threshold is not None and decode_result.compression_ratio > compression_ratio_threshold ): needs_fallback = True # too repetitive if ( logprob_threshold is not None and decode_result.avg_logprob < logprob_threshold ): needs_fallback = True # average log probability is too low if ( no_speech_threshold is not None and decode_result.no_speech_prob > no_speech_threshold and logprob_threshold is not None and decode_result.avg_logprob < logprob_threshold ): needs_fallback = False # silence if not needs_fallback: break return decode_result clip_idx = 0 seek = seek_clips[clip_idx][0] input_stride = exact_div( N_FRAMES, model.dims.n_audio_ctx ) # mel frames per output token: 2 time_precision = ( input_stride * HOP_LENGTH / SAMPLE_RATE ) # time per output token: 0.02 (seconds) all_tokens = [] all_segments = [] prompt_reset_since = 0 remaining_prompt_length = model.dims.n_text_ctx // 2 - 1 if initial_prompt is not None: initial_prompt_tokens = tokenizer.encode(" " + initial_prompt.strip()) all_tokens.extend(initial_prompt_tokens) remaining_prompt_length -= len(initial_prompt_tokens) else: initial_prompt_tokens = [] def new_segment( *, start: float, end: float, tokens: torch.Tensor, result: DecodingResult ): tokens = tokens.tolist() text_tokens = [token for token in tokens if token < tokenizer.eot] return { "seek": seek, "start": start, "end": end, "text": tokenizer.decode(text_tokens), "tokens": tokens, "temperature": result.temperature, "avg_logprob": result.avg_logprob, "compression_ratio": result.compression_ratio, "no_speech_prob": result.no_speech_prob, } # show the progress bar when verbose is False (if True, transcribed text will be printed) with tqdm.tqdm( total=content_frames, unit="frames", disable=verbose is not False ) as pbar: last_speech_timestamp = 0.0 # NOTE: This loop is obscurely flattened to make the diff readable. # A later commit should turn this into a simpler nested loop. # for seek_clip_start, seek_clip_end in seek_clips: # while seek < seek_clip_end while clip_idx < len(seek_clips): seek_clip_start, seek_clip_end = seek_clips[clip_idx] if seek < seek_clip_start: seek = seek_clip_start if seek >= seek_clip_end: clip_idx += 1 if clip_idx < len(seek_clips): seek = seek_clips[clip_idx][0] continue time_offset = float(seek * HOP_LENGTH / SAMPLE_RATE) window_end_time = float((seek + N_FRAMES) * HOP_LENGTH / SAMPLE_RATE) segment_size = min(N_FRAMES, content_frames - seek, seek_clip_end - seek) mel_segment = mel[:, seek : seek + segment_size] segment_duration = segment_size * HOP_LENGTH / SAMPLE_RATE mel_segment = pad_or_trim(mel_segment, N_FRAMES).to(model.device).to(dtype) if carry_initial_prompt: nignored = max(len(initial_prompt_tokens), prompt_reset_since) remaining_prompt = all_tokens[nignored:][-remaining_prompt_length:] decode_options["prompt"] = initial_prompt_tokens + remaining_prompt else: decode_options["prompt"] = all_tokens[prompt_reset_since:] result: DecodingResult = decode_with_fallback(mel_segment) tokens = torch.tensor(result.tokens) if no_speech_threshold is not None: # no voice activity check should_skip = result.no_speech_prob > no_speech_threshold if ( logprob_threshold is not None and result.avg_logprob > logprob_threshold ): # don't skip if the logprob is high enough, despite the no_speech_prob should_skip = False if should_skip: seek += segment_size # fast-forward to the next segment boundary continue previous_seek = seek current_segments = [] # anomalous words are very long/short/improbable def word_anomaly_score(word: dict) -> float: probability = word.get("probability", 0.0) duration = word["end"] - word["start"] score = 0.0 if probability < 0.15: score += 1.0 if duration < 0.133: score += (0.133 - duration) * 15 if duration > 2.0: score += duration - 2.0 return score def is_segment_anomaly(segment: Optional[dict]) -> bool: if segment is None or not segment["words"]: return False words = [w for w in segment["words"] if w["word"] not in punctuation] words = words[:8] score = sum(word_anomaly_score(w) for w in words) return score >= 3 or score + 0.01 >= len(words) def next_words_segment(segments: List[dict]) -> Optional[dict]: return next((s for s in segments if s["words"]), None) timestamp_tokens: torch.Tensor = tokens.ge(tokenizer.timestamp_begin) single_timestamp_ending = timestamp_tokens[-2:].tolist() == [False, True] consecutive = torch.where(timestamp_tokens[:-1] & timestamp_tokens[1:])[0] consecutive.add_(1) if len(consecutive) > 0: # if the output contains two consecutive timestamp tokens slices = consecutive.tolist() if single_timestamp_ending: slices.append(len(tokens)) last_slice = 0 for current_slice in slices: sliced_tokens = tokens[last_slice:current_slice] start_timestamp_pos = ( sliced_tokens[0].item() - tokenizer.timestamp_begin ) end_timestamp_pos = ( sliced_tokens[-1].item() - tokenizer.timestamp_begin ) current_segments.append( new_segment( start=time_offset + start_timestamp_pos * time_precision, end=time_offset + end_timestamp_pos * time_precision, tokens=sliced_tokens, result=result, ) ) last_slice = current_slice if single_timestamp_ending: # single timestamp at the end means no speech after the last timestamp. - whisper/triton_ops.py (3646 bytes)
from functools import lru_cache import numpy as np import torch try: import triton import triton.language as tl except ImportError: raise RuntimeError("triton import failed; try `pip install --pre triton`") @triton.jit def dtw_kernel( cost, trace, x, x_stride, cost_stride, trace_stride, N, M, BLOCK_SIZE: tl.constexpr ): offsets = tl.arange(0, BLOCK_SIZE) mask = offsets < M for k in range(1, N + M + 1): # k = i + j tl.debug_barrier() p0 = cost + (k - 1) * cost_stride p1 = cost + k * cost_stride p2 = cost + k * cost_stride + 1 c0 = tl.load(p0 + offsets, mask=mask) c1 = tl.load(p1 + offsets, mask=mask) c2 = tl.load(p2 + offsets, mask=mask) x_row = tl.load(x + (k - 1) * x_stride + offsets, mask=mask, other=0) cost_row = x_row + tl.minimum(tl.minimum(c0, c1), c2) cost_ptr = cost + (k + 1) * cost_stride + 1 tl.store(cost_ptr + offsets, cost_row, mask=mask) trace_ptr = trace + (k + 1) * trace_stride + 1 tl.store(trace_ptr + offsets, 2, mask=mask & (c2 <= c0) & (c2 <= c1)) tl.store(trace_ptr + offsets, 1, mask=mask & (c1 <= c0) & (c1 <= c2)) tl.store(trace_ptr + offsets, 0, mask=mask & (c0 <= c1) & (c0 <= c2)) @lru_cache(maxsize=None) def median_kernel(filter_width: int): @triton.jit def kernel( y, x, x_stride, y_stride, BLOCK_SIZE: tl.constexpr ): # x.shape[-1] == filter_width row_idx = tl.program_id(0) offsets = tl.arange(0, BLOCK_SIZE) mask = offsets < y_stride x_ptr = x + row_idx * x_stride # noqa: F841 y_ptr = y + row_idx * y_stride LOAD_ALL_ROWS_HERE # noqa: F821 BUBBLESORT_HERE # noqa: F821 tl.store(y_ptr + offsets, MIDDLE_ROW_HERE, mask=mask) # noqa: F821 kernel = triton.JITFunction(kernel.fn) new_kernel = kernel.src.replace( " LOAD_ALL_ROWS_HERE", "\n".join( [ f" row{i} = tl.load(x_ptr + offsets + {i}, mask=mask)" for i in range(filter_width) ] ), ) new_kernel = new_kernel.replace( " BUBBLESORT_HERE", "\n\n".join( [ "\n\n".join( [ "\n".join( [ f" smaller = tl.where(row{j} < row{j + 1}, row{j}, row{j + 1})", f" larger = tl.where(row{j} > row{j + 1}, row{j}, row{j + 1})", f" row{j} = smaller", f" row{j + 1} = larger", ] ) for j in range(filter_width - i - 1) ] ) for i in range(filter_width // 2 + 1) ] ), ) new_kernel = new_kernel.replace("MIDDLE_ROW_HERE", f"row{filter_width // 2}") if hasattr(kernel, "_unsafe_update_src") is True: kernel._unsafe_update_src(new_kernel) kernel.hash = None else: kernel.src = new_kernel return kernel def median_filter_cuda(x: torch.Tensor, filter_width: int): """Apply a median filter of given width along the last dimension of x""" slices = x.contiguous().unfold(-1, filter_width, 1) grid = np.prod(slices.shape[:-2]) kernel = median_kernel(filter_width) y = torch.empty_like(slices[..., 0]) BLOCK_SIZE = 1 << (y.stride(-2) - 1).bit_length() kernel[(grid,)](y, x, x.stride(-2), y.stride(-2), BLOCK_SIZE=BLOCK_SIZE) return y - whisper/utils.py (11839 bytes)
import json import os import re import sys import zlib from typing import Callable, List, Optional, TextIO system_encoding = sys.getdefaultencoding() if system_encoding != "utf-8": def make_safe(string): # replaces any character not representable using the system default encoding with an '?', # avoiding UnicodeEncodeError (https://github.com/openai/whisper/discussions/729). return string.encode(system_encoding, errors="replace").decode(system_encoding) else: def make_safe(string): # utf-8 can encode any Unicode code point, so no need to do the round-trip encoding return string def exact_div(x, y): assert x % y == 0 return x // y def str2bool(string): str2val = {"True": True, "False": False} if string in str2val: return str2val[string] else: raise ValueError(f"Expected one of {set(str2val.keys())}, got {string}") def optional_int(string): return None if string == "None" else int(string) def optional_float(string): return None if string == "None" else float(string) def compression_ratio(text) -> float: text_bytes = text.encode("utf-8") return len(text_bytes) / len(zlib.compress(text_bytes)) def format_timestamp( seconds: float, always_include_hours: bool = False, decimal_marker: str = "." ): assert seconds >= 0, "non-negative timestamp expected" milliseconds = round(seconds * 1000.0) hours = milliseconds // 3_600_000 milliseconds -= hours * 3_600_000 minutes = milliseconds // 60_000 milliseconds -= minutes * 60_000 seconds = milliseconds // 1_000 milliseconds -= seconds * 1_000 hours_marker = f"{hours:02d}:" if always_include_hours or hours > 0 else "" return ( f"{hours_marker}{minutes:02d}:{seconds:02d}{decimal_marker}{milliseconds:03d}" ) def get_start(segments: List[dict]) -> Optional[float]: return next( (w["start"] for s in segments for w in s["words"]), segments[0]["start"] if segments else None, ) def get_end(segments: List[dict]) -> Optional[float]: return next( (w["end"] for s in reversed(segments) for w in reversed(s["words"])), segments[-1]["end"] if segments else None, ) class ResultWriter: extension: str def __init__(self, output_dir: str): self.output_dir = output_dir def __call__( self, result: dict, audio_path: str, options: Optional[dict] = None, **kwargs ): audio_basename = os.path.basename(audio_path) audio_basename = os.path.splitext(audio_basename)[0] output_path = os.path.join( self.output_dir, audio_basename + "." + self.extension ) with open(output_path, "w", encoding="utf-8") as f: self.write_result(result, file=f, options=options, **kwargs) def write_result( self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs ): raise NotImplementedError class WriteTXT(ResultWriter): extension: str = "txt" def write_result( self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs ): for segment in result["segments"]: print(segment["text"].strip(), file=file, flush=True) class SubtitlesWriter(ResultWriter): always_include_hours: bool decimal_marker: str def iterate_result( self, result: dict, options: Optional[dict] = None, *, max_line_width: Optional[int] = None, max_line_count: Optional[int] = None, highlight_words: bool = False, max_words_per_line: Optional[int] = None, ): options = options or {} max_line_width = max_line_width or options.get("max_line_width") max_line_count = max_line_count or options.get("max_line_count") highlight_words = highlight_words or options.get("highlight_words", False) max_words_per_line = max_words_per_line or options.get("max_words_per_line") preserve_segments = max_line_count is None or max_line_width is None max_line_width = max_line_width or 1000 max_words_per_line = max_words_per_line or 1000 def iterate_subtitles(): line_len = 0 line_count = 1 # the next subtitle to yield (a list of word timings with whitespace) subtitle: List[dict] = [] last: float = get_start(result["segments"]) or 0.0 for segment in result["segments"]: chunk_index = 0 words_count = max_words_per_line while chunk_index < len(segment["words"]): remaining_words = len(segment["words"]) - chunk_index if max_words_per_line > len(segment["words"]) - chunk_index: words_count = remaining_words for i, original_timing in enumerate( segment["words"][chunk_index : chunk_index + words_count] ): timing = original_timing.copy() long_pause = ( not preserve_segments and timing["start"] - last > 3.0 ) has_room = line_len + len(timing["word"]) <= max_line_width seg_break = i == 0 and len(subtitle) > 0 and preserve_segments if ( line_len > 0 and has_room and not long_pause and not seg_break ): # line continuation line_len += len(timing["word"]) else: # new line timing["word"] = timing["word"].strip() if ( len(subtitle) > 0 and max_line_count is not None and (long_pause or line_count >= max_line_count) or seg_break ): # subtitle break yield subtitle subtitle = [] line_count = 1 elif line_len > 0: # line break line_count += 1 timing["word"] = "\n" + timing["word"] line_len = len(timing["word"].strip()) subtitle.append(timing) last = timing["start"] chunk_index += max_words_per_line if len(subtitle) > 0: yield subtitle if len(result["segments"]) > 0 and "words" in result["segments"][0]: for subtitle in iterate_subtitles(): subtitle_start = self.format_timestamp(subtitle[0]["start"]) subtitle_end = self.format_timestamp(subtitle[-1]["end"]) subtitle_text = "".join([word["word"] for word in subtitle]) if highlight_words: last = subtitle_start all_words = [timing["word"] for timing in subtitle] for i, this_word in enumerate(subtitle): start = self.format_timestamp(this_word["start"]) end = self.format_timestamp(this_word["end"]) if last != start: yield last, start, subtitle_text yield start, end, "".join( [ ( re.sub(r"^(\s*)(.*)$", r"\1<u>\2</u>", word) if j == i else word ) for j, word in enumerate(all_words) ] ) last = end else: yield subtitle_start, subtitle_end, subtitle_text else: for segment in result["segments"]: segment_start = self.format_timestamp(segment["start"]) segment_end = self.format_timestamp(segment["end"]) segment_text = segment["text"].strip().replace("-->", "->") yield segment_start, segment_end, segment_text def format_timestamp(self, seconds: float): return format_timestamp( seconds=seconds, always_include_hours=self.always_include_hours, decimal_marker=self.decimal_marker, ) class WriteVTT(SubtitlesWriter): extension: str = "vtt" always_include_hours: bool = False decimal_marker: str = "." def write_result( self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs ): print("WEBVTT\n", file=file) for start, end, text in self.iterate_result(result, options, **kwargs): print(f"{start} --> {end}\n{text}\n", file=file, flush=True) class WriteSRT(SubtitlesWriter): extension: str = "srt" always_include_hours: bool = True decimal_marker: str = "," def write_result( self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs ): for i, (start, end, text) in enumerate( self.iterate_result(result, options, **kwargs), start=1 ): print(f"{i}\n{start} --> {end}\n{text}\n", file=file, flush=True) class WriteTSV(ResultWriter): """ Write a transcript to a file in TSV (tab-separated values) format containing lines like: <start time in integer milliseconds>\t<end time in integer milliseconds>\t<transcript text> Using integer milliseconds as start and end times means there's no chance of interference from an environment setting a language encoding that causes the decimal in a floating point number to appear as a comma; also is faster and more efficient to parse & store, e.g., in C++. """ extension: str = "tsv" def write_result( self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs ): print("start", "end", "text", sep="\t", file=file) for segment in result["segments"]: print(round(1000 * segment["start"]), file=file, end="\t") print(round(1000 * segment["end"]), file=file, end="\t") print(segment["text"].strip().replace("\t", " "), file=file, flush=True) class WriteJSON(ResultWriter): extension: str = "json" def write_result( self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs ): json.dump(result, file) class WriteJSONL(ResultWriter): extension: str = "jsonl" def write_result( self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs ): for segment in result["segments"]: print(json.dumps(segment), file=file, flush=True) def get_writer( output_format: str, output_dir: str ) -> Callable[[dict, TextIO, dict], None]: writers = { "txt": WriteTXT, "vtt": WriteVTT, "srt": WriteSRT, "tsv": WriteTSV, "json": WriteJSON, "jsonl": WriteJSONL, } if output_format == "all": all_writers = [writer(output_dir) for writer in writers.values()] def write_all( result: dict, file: TextIO, options: Optional[dict] = None, **kwargs ): for writer in all_writers: writer(result, file, options, **kwargs) return write_all return writers[output_format](output_dir) - whisper/version.py (25 bytes)
__version__ = "20250625"