686225 - Bengali font vowel incorrect on mac osx lion

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect)

Description

[swagat]

Attachment: Screen Shot 2011-09-11 at 9.48.24 PM.png

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:6.0.2) Gecko/20100101 Firefox/6.0.2
Build ID: 20110902133214

Steps to reproduce:

using mac os x lion which has support for bengali fonts, chrome and safari display bengali fonts correctly. but firefox does not .


Actual results:

best described in screenshot


Expected results:

best described in screenshot

Comment 1

[swagat]

Attachment: the correct rendering

Comment 2

[Jonathan Kew [:jfkthame]]

Please confirm what font is being used for the Bengali text in this example. You can use the fontinfo addon[1] to determine this.

[1] https://addons.mozilla.org/en-US/firefox/addon/fontinfo/

Comment 3

[swagat]

the font being used is Bangla Sangam MN.

Comment 4

[swagat]

bug is in firefox 7 branch too.

Comment 5

[Jonathan Kew [:jfkthame]]

Note that the "correct" and "incorrect" screenshots show different fonts. Please compare results of Firefox and other browsers using the SAME font.

Comment 6

[swagat]

used the SAME font,this time. 
1)all three set to Bangla Sangam MN font: firefox- incorrect, safari,chrome- correct
2)all three set to Bangla MN font: firefox- incorrect, safari,chrome- correct

these are the only two fonts available in lion by default for bengali script

Comment 7

[swagat]

Bug still present in firefox 10. using Bangla MN font .

Comment 8

[swagat]

bug is present in firefox 11 too. Also, what is the process to mark the status of the bug as "confirmed" from "unconfirmed"

Comment 9

[Jonathan Kew [:jfkthame]]

Confirming, as I can reproduce the problem on Lion here. However, this is an OS X bug, not a Gecko bug; to demonstrate that it is not limited to Firefox, copy the example text

না না, গতকাল বাংলাদেশী বাজার থেকে কিনে আনলাম. মমতা তিস্তার জল দিলে পারত. বাংলাদেশী ইলিশ অনেক বেশি ভালো

into a new window in TextEdit.app, and use the Fonts panel to format the text in a fairly large font such as Bangla Sangam MN 24pt. Then gradually change the width of the window so that the text wraps to different widths. Note how the last word ভালো, which contains the "split vowel" ো, sometimes renders correctly and sometimes wrongly, depending on the exact content of the line.

I haven't managed to reproduce the problem in Safari, though I'm not sure why. (Given the erratic nature of it (as shown by TextEdit), perhaps they just got lucky!)

Comment 10

[swagat]

I can reproduce the problem in TextEdit. Weirdly safari/chrome never displays the same problem for a large sample of tests. I will file a bug report with apple.

Comment 11

[swagat]

TextEdit.app in os x 10.8 mountain lion now correctly renders the word ভালো  which contains the "split vowel" ো . However Firefox 15.0.1 continues to render it incorrectly.

Comment 12

[swagat]

(In reply to Jonathan Kew (:jfkthame) from comment #9)
> Confirming, as I can reproduce the problem on Lion here. However, this is an
> OS X bug, not a Gecko bug; to demonstrate that it is not limited to Firefox,
> copy the example text
> 
> না না, গতকাল বাংলাদেশী বাজার থেকে কিনে আনলাম. মমতা তিস্তার জল দিলে পারত.
> বাংলাদেশী ইলিশ অনেক বেশি ভালো
> 
> into a new window in TextEdit.app, and use the Fonts panel to format the
> text in a fairly large font such as Bangla Sangam MN 24pt. Then gradually
> change the width of the window so that the text wraps to different widths.
> Note how the last word ভালো, which contains the "split vowel" ো, sometimes
> renders correctly and sometimes wrongly, depending on the exact content of
> the line.
> 
> I haven't managed to reproduce the problem in Safari, though I'm not sure
> why. (Given the erratic nature of it (as shown by TextEdit), perhaps they
> just got lucky!)

can we confirm this as a gecko bug?using os x mavericks(10.9.1), textedit.app displays the split vowel correctly. Firefox is the only browser which displays this incorrectly.

Comment 13

[Jonathan Kew [:jfkthame]]

I don't currently have 10.9 for testing, but I still believe this must be an OS X bug. The Bangla Sangam MN font is an AAT font, so Gecko "shapes" the text by calling Core Text (an OS X component) and just renders whatever glyphs that returns. The failure to handle the split vowel properly must be happening within Core Text.

Comment 14

[swagat]

(In reply to Jonathan Kew (:jfkthame) from comment #13)
> I don't currently have 10.9 for testing, but I still believe this must be an
> OS X bug. The Bangla Sangam MN font is an AAT font, so Gecko "shapes" the
> text by calling Core Text (an OS X component) and just renders whatever
> glyphs that returns. The failure to handle the split vowel properly must be
> happening within Core Text.

Oh. The webkit based browsers(opera/chrome/safari) display correctly.

Comment 15

[Jonathan Kew [:jfkthame]]

(In reply to swagat from comment #14)
> (In reply to Jonathan Kew (:jfkthame) from comment #13)
> > I don't currently have 10.9 for testing, but I still believe this must be an
> > OS X bug. The Bangla Sangam MN font is an AAT font, so Gecko "shapes" the
> > text by calling Core Text (an OS X component) and just renders whatever
> > glyphs that returns. The failure to handle the split vowel properly must be
> > happening within Core Text.
> 
> Oh. The webkit based browsers(opera/chrome/safari) display correctly.

The erratic behavior of TextEdit on 10.7 suggested to me that the problem is very sensitive to exactly how the underlying AAT shaping code gets used in a particular application - sometimes it worked correctly, sometimes it failed.

Interestingly, though, I can no longer seem to reproduce the behavior in TextEdit on 10.7. (Perhaps something has been changed during a system update?)

It does seem possible that there's a subtle bug in how Gecko interfaces with Core Text, and that's leading to the incorrect rendering here. It'd be a bit surprising given that TextEdit used to suffer (intermittently) from the same issue, but it'd be worth trying to trace more carefully exactly what happens in gfxCoreTextShaper during the shaping of the problem sequences.

Comment 16

[Jonathan Kew [:jfkthame]]

Test string: U+0995,U+0995,U+09CB using Bangla Sangam MN.ttf

(A) Using standalone hb-view tool with debugging printfs added:

CFShow(attr_string) immediately before the call to CTLineCreateWithAttributedString:

\u0995\u0995\u09cb 0x7fa0e50070f0 {NSFont=CTFont <name: BanglaSangamMN, size: 2048.000000, matrix: 0x0>
CTFontDescriptor <attributes: <CFBasicHash 0x7fa0e50074f0 [0x7fff7c0c5fa0]>{type = mutable dict, count = 1,
entries =>
	2 : <CFString 0x7fff7be0ab38 [0x7fff7c0c5fa0]>{contents = "NSFontNameAttribute"} = <CFString 0x7fa0e5007270 [0x7fff7c0c5fa0]>{contents = "BanglaSangamMN"}
}
>} Len 3

Resulting glyph run in the CTLine (printing glyph IDs and original string indices):

Glyphs: 128@0  168@2     128@1  602@2
        bn_ka  bn_ekaar  bn_ka  bn_aakaar

Note how the character U+09CB at index 2 has been "split" into two glyphs, the bn_ekaar that is moved before the consonant, and bn_aakaar that remains in the original post-consonant position.


(B) Within Firefox, adding debugging print statements in gfxCoreTextShaper.cpp:

CFShow(attrStringObj) immediately before the call to CTLineCreateWithAttributedString:

\u0995\u0995\u09cb 0x12340eac0 {NSFont=CTFont <name: BanglaSangamMN, size: 500.000000, matrix: 0x0>
CTFontDescriptor <attributes: <CFBasicHash 0x12268ad00 [0x7fff7c0c5fa0]>{type = mutable dict, count = 1,
entries =>
	2 : <CFString 0x7fff7be0ab38 [0x7fff7c0c5fa0]>{contents = "NSFontNameAttribute"} = <CFString 0x1189369e0 [0x7fff7c0c5fa0]>{contents = "BanglaSangamMN"}
}
>} Len 3

Resulting glyph run in the CTLine:

Glyphs: 128@0  128@1  170@2     602@2
        bn_ka  bn_ka  bn_okaar  bn_aakaar

Note that U+09CB (matra o) has been "split" to produce the extra bn_aakaar glyph for the right-hand part, but the bn_okaar glyph has not been changed to bn_ekaar, which would be the appropriate glyph for the left-hand part of the vowel. And therefore it has also not been reordered to the pre-consonant position.


Conclusion: given the exact same Unicode string, and the same AAT font, CTLineCreateWithAttributedString is sometimes failing to shape the split vowel U+09CB properly. It looks like it's specifically the feature "Stand Alone Matras" in the Bangla font that isn't being applied correctly. There's no significant difference in the CFAttributedString parameters being passed to the Core Text function; why is it behaving inconsistently?

I suspect there may be something like an uninitialized variable used within the Core Text shaping code, so that its behavior depends on what happens to be in some particular register or memory location when it is called.

Comment 18

[Jonathan Kew [:jfkthame]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a01a84573c61

Comment 19

[Jonathan Kew [:jfkthame]]

I've just been looking into this again, and concluded that (at least on OS X 10.9), the issue here is that the Bangla MN and Bangla Sangam MN fonts (as well as their Kannada counterparts, see bug 728557 and bug 953231) depends on the "Line Initial Smart Swash" feature being enabled. If this feature is turned off, the shaping of certain split vowels fails.

In Gecko, we shape with these line-edge features disabled because they can affect measurement, and at the time of shaping we don't yet know where the line breaks will be.

This is a font bug, IMO: line-initial (and -final) swashes are a typographic option that should not affect the basic shaping needed for linguistically-correct rendering. Nevertheless, given that this applies to the default fonts Apple ships for these scripts, I think we should try to work around the problem. Patch coming.

Comment 20

[Jonathan Kew [:jfkthame]]

Attachment: Work around buggy AAT fonts for Bengali and Kannada scripts

Try job in progress: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bba4e84d2923. Will flag for review after checking test results.

Comment 21

[Jonathan Kew [:jfkthame]]

Attachment: Work around buggy AAT fonts for Bengali and Kannada scripts

Comment 22

[Jonathan Kew [:jfkthame]]

FTR, I reported the problem with the abuse of registered smart-swash feature selectors in Bangla and Kannada fonts as rdar://20257092.

Comment 23

[John Daggett (:jtd)]

Comment on attachment 8580716 [details] [diff] [review]
Work around buggy AAT fonts for Bengali and Kannada scripts

Review of attachment 8580716 [details] [diff] [review]:
-----------------------------------------------------------------

r+ Unfortunate that we have to clutter up the code like this.

If there's an easy way to make a reftest, that would be nice to include.

>+    if (IsBuggyIndicScript(aScript)) {
>+        // To work around buggy Indic AAT fonts shipped with OS X,
>+        // we re-enable the Line Initial Smart Swashes feature that is needed
>+        // for "split vowels" to work in at least Bengali and Kannada fonts.
>+        // See bug reports such as 686225, 728557, 953231, 1145515
>+        tempCTFont =
>+            CreateCTFontWithFeatures(::CTFontGetSize(mCTFont),
>+                                     aShapedText->DisableLigatures()
>+                                         ? GetIndicDisableLigaturesDescriptor()
>+                                         : GetIndicFeaturesDescriptor());

I think it would be useful to list here the specific fonts that appear to contain this bug.

Comment 24

[Jonathan Kew [:jfkthame]]

Attachment: Reftest for Indic shaping with buggy OS X (AAT) fonts

It's difficult to positively reftest the precise shaping of a bunch of Indic text, but we can at least test that this specific bug no longer occurs. The examples here fail on current trunk, and pass with the patch; they should harmlessly pass on any other platform, too, regardless of available fonts.

Comment 25

[Jonathan Kew [:jfkthame]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=09f3f9aad8a1

Comment 26

[Jonathan Kew [:jfkthame]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/23e60e2a05a5
https://hg.mozilla.org/integration/mozilla-inbound/rev/bfbca6c64c1b

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/23e60e2a05a5
https://hg.mozilla.org/mozilla-central/rev/bfbca6c64c1b

Comment 28

[dipanjan]

Jonathan, thanks for figuring out this workaround. 

For the benefit of any Firefox ESR users (I think this fix won't make it into the next ESR which is based on Firefox 38), let me just mention that one can also download and install Google's "Noto Sans Bengali" font, which is rendered properly in Firefox. This can be found at 
https://www.google.com/get/noto/
Once the font is installed, go to the Content tab in Firefox's Preferences dialog box, click the "Advanced..." button under Fonts & Colors, select Bengali under the "Fonts for" dropdown, and replace any references to Bangla MN Fonts with the Noto Sans Bengali font you just installed.