Issue with a table cell containing multiple lines of text

[codePigPig]

Sorry to interrupt everyone, but I have a question and I would really appreciate your help. When dealing with tables in a PDF where a cell contains multiple lines of text, I'm having difficulty extracting the data in the desired structure. I'm not sure if I'm doing something wrong, as I've just started learning pdfplumber. I would greatly appreciate any guidance or suggestions from all of you. Thank you so much.

This is my form:
https://ibb.co/ZNQdxQH

Data read by pdfplumber：
[
['主要研发项目名称', '项目目的', '项目进展', '拟达到的目标', '预计对公司未来发展的影响'],
['', '全新序列首款产品，提\n升市场竞争力', '', '开发全新电动车型，按计划', '开发新产品，提升市场竞争'],
['高级舒适轿车开发', None, '完成年度开发任务', None, None], [None, None, None, '上市销售', '力'],
['', None, '', None, None],
['全新一代插电混动车型', '开发新产品，提升市场\n竞争力', '', '开发全新一代插电混动车', '开发新产品，提升市场竞争'],
[None, None, '完成年度开发任务', None, None],
['开发', None, None, '型，按计划上市销售', '力'],
[None, None, '', None, None],
['全新一代全电数字轿车', '长安新能源首款纯电\n平台战略车型', '', '开发全新电动车型，按计划', '开发新产品，提升市场竞争'],
[None, None, '完成年度开发任务', None, None],
['开发', None, None, '上市销售', '力'], [None, None, '', None, None],
['智能电动数字化平台开', '打造领先的软硬件平\n台', '', '完成主体功能 100%开发，', ''],
[None, None, '完成年度研发任务', None, '新汽车转型升级'],
['发', None, None, '技术状态锁定', None], [None, None, '', None, ''],
['', '突破技术瓶颈，强化电\n动化、智能化产品占\n位，支撑公司中大型市\n场产品开发', '', '', ''],
[None, None, None, None, '加速向智能低碳出行科技公'],
['全新一代纯电智能整车', None, None, '完成平台开发，实现全面平', None],
[None, None, '完成年度研发任务', None, '司转型，支撑“新汽车 新生'],
['平台开发', None, None, '台化、智能化、电气化', None],
[None, None, '', None, '态”的发展策略'],
['', None, None, '', None],
[None, None, None, None, ''],
['多动力兼容架构整车平', '长安兼容平台架构，支\n撑多动力产品', '', '拓展优化系统构架，实现多', '丰富公司产品动力选择，提'],
[None, None, '完成年度研发任务', None, None],
['台开发', None, None, '动力平台共平台开发', '升开发效率和成本'],
[None, None, '', None, None]]

The results I want：
['主要研发项目名称', '项目目的', '项目进展', '拟达到的目标', '预计对公司未来发展的影响'],
['高级舒适轿车开发', '全新序列首款产品，提升市场竞争力', '完成年度开发任务', '开发全新电动车型，按计划 上市销售', '开发新产品，提升市场竞争 力']
.....

problem：
1、 After the table extracts data, when there are multiple lines of text in one place, the content cannot be contiguous.

I look forward to your reply. Thank you.

[samkit-jain]

Hi @codePigPig Could you please attach the PDF? The link you have provided isn't working

[codePigPig]

Thank you for your reply
Sorry, I didn't realize that the link was broken before.

This is the test PDF document I used：
http://rvlt9r7b8.sabkt.gdipper.com/pdftest01.pdf?e=1685759184&token=0nt2qr2wh1RrjapAKgb3Fm9_f_dqDs1RdXxCokIc:Lzcdt-v5eJjs4m1bN4uWpO6dvmU=

Remarks：
Because I am using free cloud storage, the link address for all PDF documents is only valid for 24 hours. I'm very sorry about that.

[samkit-jain]

I am getting {"error":"token out of date"} now. Looks like the link expired. You can also upload your file here. When replying to a message, you'll notice that at the bottom there is an option for attaching files. Use that please to share the file.

[codePigPig]

I'm very sorry that I haven't been using GitHub frequently, so I'm not very familiar with it. Thank you for teaching me how to use it.

This is my test file：
pdftest01.pdf

[samkit-jain]

Thanks for sharing the PDF. If you extract tables with the default lines-lines settings, you'll get the data as

As you can see, there are a few lines here that are not visible in the PDF. To remove those, you can apply a filter as below

import pdfplumber


def keep_visible_lines(obj):
    """
    If the object is a ``rect`` type, keep it only if the lines are visible.

    A visible line is the one having ``non_stroking_color`` not null.
    """
    if obj['object_type'] == 'rect':
        return obj['non_stroking_color'] is None
    return True


pdf = pdfplumber.open("file.pdf")
page0 = pdf.pages[0]
page0 = page0.filter(keep_visible_lines)

ts = {"vertical_strategy": "lines", "horizontal_strategy": "lines"}

# This code saves the debug visual output.
im = page0.to_image(resolution=200)
im.reset().debug_tablefinder(ts)
im.save("image.png", format="PNG")

# Extract the tables.
tables = page0.extract_tables(table_settings=ts)
for table in tables:
    print()
    for row in table:
        print(row)

It will give you the output as

['3', '第三名', '3,463,898,449.23', '3.59%']
['4', '第四名', '1,683,131,736.33', '1.75%']
['5', '第五名', '1,439,755,918.75', '1.49%']
['合计', '--', '17,209,662,321.50', '17.85%']

['', '2022年', '2021年', '同比增减', '重大变动说明']
['销售费用', '5,138,270,162.95', '4,645,654,234.59', '10.60%', '']
['管理费用', '3,532,458,198.76', '3,499,654,508.49', '0.94%', '']
['财务费用', '-1,017,305,743.08', '-780,923,197.18', '30.27%', '由于公司报告期内月均货币资金余\n额同比增加，金融机构存款利息收入\n增加']
['研发费用', '4,315,444,864.76', '3,515,029,178.76', '22.77%', '']

['主要研发项目名称', '项目目的', '项目进展', '拟达到的目标', '预计对公司未来发展的影响']
['高级舒适轿车开发', '全新序列首款产品，提\n升市场竞争力', '完成年度开发任务', '开发全新电动车型，按计划\n上市销售', '开发新产品，提升市场竞争\n力']
['全新一代插电混动车型\n开发', '开发新产品，提升市场\n竞争力', '完成年度开发任务', '开发全新一代插电混动车\n型，按计划上市销售', '开发新产品，提升市场竞争\n力']
['全新一代全电数字轿车\n开发', '长安新能源首款纯电\n平台战略车型', '完成年度开发任务', '开发全新电动车型，按计划\n上市销售', '开发新产品，提升市场竞争\n力']
['智能电动数字化平台开\n发', '打造领先的软硬件平\n台', '完成年度研发任务', '完成主体功能 100%开发，\n技术状态锁定', '新汽车转型升级']
['全新一代纯电智能整车\n平台开发', '突破技术瓶颈，强化电\n动化、智能化产品占\n位，支撑公司中大型市\n场产品开发', '完成年度研发任务', '完成平台开发，实现全面平\n台化、智能化、电气化', '加速向智能低碳出行科技公\n司转型，支撑“新汽车 新生\n态”的发展策略']
['多动力兼容架构整车平\n台开发', '长安兼容平台架构，支\n撑多动力产品', '完成年度研发任务', '拓展优化系统构架，实现多\n动力平台共平台开发', '丰富公司产品动力选择，提\n升开发效率和成本']

['', '2022年', '2021年', '变动比例']
['研发人员数量（人）', '7,899', '7,269', '8.67%']
['研发人员数量占比', '18.42%', '17.13%', '1.29%']

and your visual output reflects the same as well

Please note that the method keep_visible_lines(...) is specific to this PDF and may not work on other PDFs.

[codePigPig]

Wow, that's amazing. This is exactly what I want for the data structure. Thank you very much for your response and guidance.

[cmdlineluser]

It looks similar to the issue discussed here:

#839 (reply in thread)

def reject_2d_rects(obj):
    is_rect = obj["object_type"] == "rect"
    is_thin = obj["width"] < 1 or obj["height"] < 1
    return not (is_rect and not is_thin)

filtered = page.filter(reject_2d_rects)

>>> filtered.extract_tables()[-2]
[['主要研发项目名称', '项目目的', '项目进展', '拟达到的目标', '预计对公司未来发展的影响'],
 ['高级舒适轿车开发',
  '全新序列首款产品，提\n升市场竞争力',
  '完成年度开发任务',
  '开发全新电动车型，按计划\n上市销售',
  '开发新产品，提升市场竞争\n力'],
 ['全新一代插电混动车型\n开发',
  '开发新产品，提升市场\n竞争力',
  '完成年度开发任务',
  '开发全新一代插电混动车\n型，按计划上市销售',
  '开发新产品，提升市场竞争\n力'],
 ['全新一代全电数字轿车\n开发',
  '长安新能源首款纯电\n平台战略车型',
  '完成年度开发任务',
  '开发全新电动车型，按计划\n上市销售',
  '开发新产品，提升市场竞争\n力'],
 ['智能电动数字化平台开\n发',
  '打造领先的软硬件平\n台',
  '完成年度研发任务',
  '完成主体功能 100%开发，\n技术状态锁定',
  '新汽车转型升级'],
 ['全新一代纯电智能整车\n平台开发',
  '突破技术瓶颈，强化电\n动化、智能化产品占\n位，支撑公司中大型市\n场产品开发',
  '完成年度研发任务',
  '完成平台开发，实现全面平\n台化、智能化、电气化',
  '加速向智能低碳出行科技公\n司转型，支撑“新汽车 新生\n态”的发展策略'],
 ['多动力兼容架构整车平\n台开发',
  '长安兼容平台架构，支\n撑多动力产品',
  '完成年度研发任务',
  '拓展优化系统构架，实现多\n动力平台共平台开发',
  '丰富公司产品动力选择，提\n升开发效率和成本']]

[codePigPig]

Thank you very much for your response and guidance.
I tried the method you provided, and it works. Thank you very much for your help.

[CuriousTank]

这看起来和这里讨论的问题很相似：

#839（回复此帖）

def reject_2d_rects(obj):
    is_rect = obj["object_type"] == "rect"
    is_thin = obj["width"] < 1 or obj["height"] < 1
    return not (is_rect and not is_thin)

filtered = page.filter(reject_2d_rects)

>>> filtered.extract_tables()[-2]
[['主要研发项目名称', '项目目的', '项目进展', '拟达到的目标', '预计对公司未来发展的影响'],
 ['高级舒适轿车开发',
  '全新序列首款产品，提\n升市场竞争力',
  '完成年度开发任务',
  '开发全新电动车型，按计划\n上市销售',
  '开发新产品，提升市场竞争\n力'],
 ['全新一代插电混动车型\n开发',
  '开发新产品，提升市场\n竞争力',
  '完成年度开发任务',
  '开发全新一代插电混动车\n型，按计划上市销售',
  '开发新产品，提升市场竞争\n力'],
 ['全新一代全电数字轿车\n开发',
  '长安新能源首款纯电\n平台战略车型',
  '完成年度开发任务',
  '开发全新电动车型，按计划\n上市销售',
  '开发新产品，提升市场竞争\n力'],
 ['智能电动数字化平台开\n发',
  '打造领先的软硬件平\n台',
  '完成年度研发任务',
  '完成主体功能 100%开发，\n技术状态锁定',
  '新汽车转型升级'],
 ['全新一代纯电智能整车\n平台开发',
  '突破技术瓶颈，强化电\n动化、智能化产品占\n位，支撑公司中大型市\n场产品开发',
  '完成年度研发任务',
  '完成平台开发，实现全面平\n台化、智能化、电气化',
  '加速向智能低碳出行科技公\n司转型，支撑“新汽车 新生\n态”的发展策略'],
 ['多动力兼容架构整车平\n台开发',
  '长安兼容平台架构，支\n撑多动力产品',
  '完成年度研发任务',
  '拓展优化系统构架，实现多\n动力平台共平台开发',
  '丰富公司产品动力选择，提\n升开发效率和成本']]

Filtering the small box is a commonly used method. Thanks for pointing out where to perform the filtering operation and saved me a lot of time. Thank you