PostgreSQL
Check-in [ee787097fc]
Not logged in

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Make sure chr(int) can't create invalid UTF8 sequences. Several years ago we changed chr(int) so that if the database encoding is UTF8, it would interpret its argument as a Unicode code point and expand it into the appropriate multibyte sequence. However, we weren't sufficiently careful about checking validity of the input. According to RFC3629, UTF8 disallows code points above U+10FFFF (note that the predecessor standard RFC2279 was more liberal). Also, both versions of the UTF8 spec agree that Unicode surrogate-pair codes should never appear in UTF8. Because our encoding validity checks follow RFC3629, our failure to enforce these restrictions in chr() means it could be used to produce text strings that will be rejected when the database is dumped and reloaded. To ensure consistency with the input functions, let's actually apply pg_utf8_islegal() to the proposed output of chr(). Per discussion, this seems like too much of a behavioral change to back-patch, but it's not too late to squeeze it into 9.4.
Timelines: family | ancestors | descendants | both | trunk | WIN32_DEV | REL9_0_ALPHA4_BRANCH
Files: files | file ages | folders
SHA1:ee787097fc8d667e95ebee54ee53abf2b151b9d2
User & Date: tgl@sss.pgh.pa.us 2014-05-16 20:51:28
Context
2014-05-17
10:55
Update README, we don't do post-recovery cleanup actions anymore. transam/README explained how B-tree incomplete splits were tracked and fixed after recovery, as an example of handling complex action... Leaf check-in: d250ce3c3c user: heikki.linnakangas@iki.fi tags: trunk, WIN32_DEV, REL9_0_ALPHA4_BRANCH
2014-05-16
20:51
Make sure chr(int) can't create invalid UTF8 sequences. Several years ago we changed chr(int) so that if the database encoding is UTF8, it would interpret its argument as a Unicode code point and exp... check-in: ee787097fc user: tgl@sss.pgh.pa.us tags: trunk, WIN32_DEV, REL9_0_ALPHA4_BRANCH
19:29
Suppress some more valgrind whining about btree_gist. A couple of functions didn't bother to zero out pad bytes in datums that would ultimately go to disk. Harmless, but valgrind doesn't know that. check-in: e846fca3d6 user: tgl@sss.pgh.pa.us tags: trunk, WIN32_DEV, REL9_0_ALPHA4_BRANCH
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to src/backend/utils/adt/oracle_compat.c.

928
929
930
931
932
933
934
935
936

937



938
939
940
941
942
943
944
945
...
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
...
967
968
969
970
971
972
973










974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
	text	   *result;
	int			encoding = GetDatabaseEncoding();

	if (encoding == PG_UTF8 && cvalue > 127)
	{
		/* for Unicode we treat the argument as a code point */
		int			bytes;
		char	   *wch;


		/* We only allow valid Unicode code points */



		if (cvalue > 0x001fffff)
			ereport(ERROR,
					(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
					 errmsg("requested character too large for encoding: %d",
							cvalue)));

		if (cvalue > 0xffff)
			bytes = 4;
................................................................................
		else if (cvalue > 0x07ff)
			bytes = 3;
		else
			bytes = 2;

		result = (text *) palloc(VARHDRSZ + bytes);
		SET_VARSIZE(result, VARHDRSZ + bytes);
		wch = VARDATA(result);

		if (bytes == 2)
		{
			wch[0] = 0xC0 | ((cvalue >> 6) & 0x1F);
			wch[1] = 0x80 | (cvalue & 0x3F);;
		}
		else if (bytes == 3)
................................................................................
		{
			wch[0] = 0xF0 | ((cvalue >> 18) & 0x07);
			wch[1] = 0x80 | ((cvalue >> 12) & 0x3F);
			wch[2] = 0x80 | ((cvalue >> 6) & 0x3F);
			wch[3] = 0x80 | (cvalue & 0x3F);
		}











	}

	else
	{
		bool		is_mb;

		/*
		 * Error out on arguments that make no sense or that we can't validly
		 * represent in the encoding.
		 */

		if (cvalue == 0)
			ereport(ERROR,
					(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
					 errmsg("null character not permitted")));

		is_mb = pg_encoding_max_length(encoding) > 1;

		if ((is_mb && (cvalue > 127)) || (!is_mb && (cvalue > 255)))
			ereport(ERROR,
					(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
					 errmsg("requested character too large for encoding: %d",
							cvalue)));


		result = (text *) palloc(VARHDRSZ + 1);
		SET_VARSIZE(result, VARHDRSZ + 1);
		*VARDATA(result) = (char) cvalue;
	}

	PG_RETURN_TEXT_P(result);







|

>
|
>
>
>
|







 







|







 







>
>
>
>
>
>
>
>
>
>

<








<












<







928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
...
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
...
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988

989
990
991
992
993
994
995
996

997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013
1014
1015
	text	   *result;
	int			encoding = GetDatabaseEncoding();

	if (encoding == PG_UTF8 && cvalue > 127)
	{
		/* for Unicode we treat the argument as a code point */
		int			bytes;
		unsigned char *wch;

		/*
		 * We only allow valid Unicode code points; per RFC3629 that stops at
		 * U+10FFFF, even though 4-byte UTF8 sequences can hold values up to
		 * U+1FFFFF.
		 */
		if (cvalue > 0x0010ffff)
			ereport(ERROR,
					(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
					 errmsg("requested character too large for encoding: %d",
							cvalue)));

		if (cvalue > 0xffff)
			bytes = 4;
................................................................................
		else if (cvalue > 0x07ff)
			bytes = 3;
		else
			bytes = 2;

		result = (text *) palloc(VARHDRSZ + bytes);
		SET_VARSIZE(result, VARHDRSZ + bytes);
		wch = (unsigned char *) VARDATA(result);

		if (bytes == 2)
		{
			wch[0] = 0xC0 | ((cvalue >> 6) & 0x1F);
			wch[1] = 0x80 | (cvalue & 0x3F);;
		}
		else if (bytes == 3)
................................................................................
		{
			wch[0] = 0xF0 | ((cvalue >> 18) & 0x07);
			wch[1] = 0x80 | ((cvalue >> 12) & 0x3F);
			wch[2] = 0x80 | ((cvalue >> 6) & 0x3F);
			wch[3] = 0x80 | (cvalue & 0x3F);
		}

		/*
		 * The preceding range check isn't sufficient, because UTF8 excludes
		 * Unicode "surrogate pair" codes.  Make sure what we created is valid
		 * UTF8.
		 */
		if (!pg_utf8_islegal(wch, bytes))
			ereport(ERROR,
					(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
					 errmsg("requested character not valid for encoding: %d",
							cvalue)));
	}

	else
	{
		bool		is_mb;

		/*
		 * Error out on arguments that make no sense or that we can't validly
		 * represent in the encoding.
		 */

		if (cvalue == 0)
			ereport(ERROR,
					(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
					 errmsg("null character not permitted")));

		is_mb = pg_encoding_max_length(encoding) > 1;

		if ((is_mb && (cvalue > 127)) || (!is_mb && (cvalue > 255)))
			ereport(ERROR,
					(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
					 errmsg("requested character too large for encoding: %d",
							cvalue)));


		result = (text *) palloc(VARHDRSZ + 1);
		SET_VARSIZE(result, VARHDRSZ + 1);
		*VARDATA(result) = (char) cvalue;
	}

	PG_RETURN_TEXT_P(result);