Convert Unicode Symbols & Punctuation to ASCII using ColdFusion/Java

symbolsToASCII is a ColdFusion UDF (user-defined function) to convert Unicode symbols and punctuation to ASCII7. I was previously using ConvertSpecialChars from CFLib, but it didn't include enough mapped characters.

I found some documentation from NIH's Lexical Systems Group website that documented their approach to "Map Symbols & Punctuation to ASCII". They state that "converting Unicode punctuation and symbols to ASCII punctuation and symbols is imperative in NLP for preserving the original documents. Their java code implementation is simply "perform mapping if the character is in the punctuation & symbols mapping table".

Their approach makes a lot of sense. When I'm performing a search using a SQL query or a Verity collection, the HTML5 input field doesn't auto-corrupt "dumb quotes" to “smart quotes” like Microsoft Word does. If stored content has characters that are HTML-encoded, wouldn't extra logic be required to account for potential substitutions containing high ASCII characters as well as ‘, ’, “ and ”?

Usage: symbolsToASCII(required string inputString)

<cfset testString = '#CHR(8220)#I don#CHR(8217)#t know what you mean by #CHR(8216)#glory,#CHR(8217)# #CHR(8221)# Alice said.'>

<cfoutput>
<textarea style="width:95%; height:300px;">
Original: #TestString#

symbolsToASCII: #symbolsToASCII(testString)#
</textarea>

Try it online at TryCF.com

https://trycf.com/gist/6f35220d47caa7fdbf75eb884ff1cec7

Source code

Symbols and Punctuation Default Mapping Table (credit)